2023-04-24 08:58:08,094 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5 2023-04-24 08:58:08,106 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-24 08:58:08,118 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-24 08:58:08,118 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/cluster_34f00b0f-ffba-bb69-e77b-b55f002d7f8b, deleteOnExit=true 2023-04-24 08:58:08,118 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-24 08:58:08,119 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/test.cache.data in system properties and HBase conf 2023-04-24 08:58:08,119 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/hadoop.tmp.dir in system properties and HBase conf 2023-04-24 08:58:08,120 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/hadoop.log.dir in system properties and HBase conf 2023-04-24 08:58:08,120 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-24 08:58:08,120 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-24 08:58:08,121 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-24 08:58:08,209 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-24 08:58:08,507 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-24 08:58:08,511 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-24 08:58:08,511 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-24 08:58:08,512 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-24 08:58:08,512 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-24 08:58:08,512 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-24 08:58:08,512 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-24 08:58:08,513 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-24 08:58:08,513 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-24 08:58:08,513 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-24 08:58:08,513 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/nfs.dump.dir in system properties and HBase conf 2023-04-24 08:58:08,514 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/java.io.tmpdir in system properties and HBase conf 2023-04-24 08:58:08,514 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-24 08:58:08,514 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-24 08:58:08,514 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-24 08:58:08,956 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-24 08:58:08,960 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-24 08:58:09,345 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-24 08:58:09,523 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-24 08:58:09,537 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-24 08:58:09,574 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-24 08:58:09,601 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/java.io.tmpdir/Jetty_localhost_localdomain_39955_hdfs____.7sqmr7/webapp 2023-04-24 08:58:09,722 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:39955 2023-04-24 08:58:09,759 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-24 08:58:09,760 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-24 08:58:10,283 WARN [Listener at localhost.localdomain/46389] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-24 08:58:10,402 WARN [Listener at localhost.localdomain/46389] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-24 08:58:10,421 WARN [Listener at localhost.localdomain/46389] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-24 08:58:10,429 INFO [Listener at localhost.localdomain/46389] log.Slf4jLog(67): jetty-6.1.26 2023-04-24 08:58:10,435 INFO [Listener at localhost.localdomain/46389] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/java.io.tmpdir/Jetty_localhost_46221_datanode____.98af6y/webapp 2023-04-24 08:58:10,542 INFO [Listener at localhost.localdomain/46389] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46221 2023-04-24 08:58:10,835 WARN [Listener at localhost.localdomain/44291] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-24 08:58:11,598 DEBUG [Listener at localhost.localdomain/44291] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5 2023-04-24 08:58:11,601 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3bc8bf0cbda709a2: Processing first storage report for DS-65d8d798-0a6a-484f-9e53-119be27d2c27 from datanode 23f6c51a-5397-46cb-8527-136c6ed29987 2023-04-24 08:58:11,603 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3bc8bf0cbda709a2: from storage DS-65d8d798-0a6a-484f-9e53-119be27d2c27 node DatanodeRegistration(127.0.0.1:37329, datanodeUuid=23f6c51a-5397-46cb-8527-136c6ed29987, infoPort=44243, infoSecurePort=0, ipcPort=44291, storageInfo=lv=-57;cid=testClusterID;nsid=459647377;c=1682326689016), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-24 08:58:11,603 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3bc8bf0cbda709a2: Processing first storage report for DS-6b2e4bd3-1119-440f-bec0-60a27126bc44 from datanode 23f6c51a-5397-46cb-8527-136c6ed29987 2023-04-24 08:58:11,603 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3bc8bf0cbda709a2: from storage DS-6b2e4bd3-1119-440f-bec0-60a27126bc44 node DatanodeRegistration(127.0.0.1:37329, datanodeUuid=23f6c51a-5397-46cb-8527-136c6ed29987, infoPort=44243, infoSecurePort=0, ipcPort=44291, storageInfo=lv=-57;cid=testClusterID;nsid=459647377;c=1682326689016), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-24 08:58:11,661 INFO [Listener at localhost.localdomain/44291] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/cluster_34f00b0f-ffba-bb69-e77b-b55f002d7f8b/zookeeper_0, clientPort=63978, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/cluster_34f00b0f-ffba-bb69-e77b-b55f002d7f8b/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/cluster_34f00b0f-ffba-bb69-e77b-b55f002d7f8b/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-24 08:58:11,676 INFO [Listener at localhost.localdomain/44291] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63978 2023-04-24 08:58:11,683 INFO [Listener at localhost.localdomain/44291] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 08:58:11,685 INFO [Listener at localhost.localdomain/44291] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 08:58:12,304 INFO [Listener at localhost.localdomain/44291] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f with version=8 2023-04-24 08:58:12,305 INFO [Listener at localhost.localdomain/44291] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/hbase-staging 2023-04-24 08:58:12,619 INFO [Listener at localhost.localdomain/44291] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-24 08:58:12,993 INFO [Listener at localhost.localdomain/44291] client.ConnectionUtils(127): master/jenkins-hbase16:0 server-side Connection retries=45 2023-04-24 08:58:13,017 INFO [Listener at localhost.localdomain/44291] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-24 08:58:13,018 INFO [Listener at localhost.localdomain/44291] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-24 08:58:13,018 INFO [Listener at localhost.localdomain/44291] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-24 08:58:13,018 INFO [Listener at localhost.localdomain/44291] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-24 08:58:13,018 INFO [Listener at localhost.localdomain/44291] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-24 08:58:13,133 INFO [Listener at localhost.localdomain/44291] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-24 08:58:13,212 DEBUG [Listener at localhost.localdomain/44291] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-24 08:58:13,291 INFO [Listener at localhost.localdomain/44291] ipc.NettyRpcServer(120): Bind to /188.40.62.62:40429 2023-04-24 08:58:13,300 INFO [Listener at localhost.localdomain/44291] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 08:58:13,302 INFO [Listener at localhost.localdomain/44291] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 08:58:13,318 INFO [Listener at localhost.localdomain/44291] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40429 connecting to ZooKeeper ensemble=127.0.0.1:63978 2023-04-24 08:58:13,396 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:404290x0, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-24 08:58:13,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40429-0x101ba998db50000 connected 2023-04-24 08:58:13,469 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-24 08:58:13,471 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-24 08:58:13,476 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-24 08:58:13,485 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40429 2023-04-24 08:58:13,486 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40429 2023-04-24 08:58:13,486 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40429 2023-04-24 08:58:13,487 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40429 2023-04-24 08:58:13,487 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40429 2023-04-24 08:58:13,493 INFO [Listener at localhost.localdomain/44291] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f, hbase.cluster.distributed=false 2023-04-24 08:58:13,551 INFO [Listener at localhost.localdomain/44291] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=45 2023-04-24 08:58:13,551 INFO [Listener at localhost.localdomain/44291] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-24 08:58:13,551 INFO [Listener at localhost.localdomain/44291] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-24 08:58:13,551 INFO [Listener at localhost.localdomain/44291] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-24 08:58:13,551 INFO [Listener at localhost.localdomain/44291] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-24 08:58:13,551 INFO [Listener at localhost.localdomain/44291] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-24 08:58:13,555 INFO [Listener at localhost.localdomain/44291] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-24 08:58:13,558 INFO [Listener at localhost.localdomain/44291] ipc.NettyRpcServer(120): Bind to /188.40.62.62:42837 2023-04-24 08:58:13,560 INFO [Listener at localhost.localdomain/44291] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-24 08:58:13,565 DEBUG [Listener at localhost.localdomain/44291] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-24 08:58:13,565 INFO [Listener at localhost.localdomain/44291] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 08:58:13,568 INFO [Listener at localhost.localdomain/44291] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 08:58:13,569 INFO [Listener at localhost.localdomain/44291] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42837 connecting to ZooKeeper ensemble=127.0.0.1:63978 2023-04-24 08:58:13,575 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:428370x0, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-24 08:58:13,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42837-0x101ba998db50001 connected 2023-04-24 08:58:13,576 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-24 08:58:13,578 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-24 08:58:13,579 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-24 08:58:13,579 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42837 2023-04-24 08:58:13,580 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42837 2023-04-24 08:58:13,580 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42837 2023-04-24 08:58:13,581 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42837 2023-04-24 08:58:13,581 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42837 2023-04-24 08:58:13,583 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 08:58:13,601 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-24 08:58:13,602 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 08:58:13,625 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-24 08:58:13,625 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-24 08:58:13,625 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 08:58:13,626 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-24 08:58:13,628 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase16.apache.org,40429,1682326692433 from backup master directory 2023-04-24 08:58:13,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-24 08:58:13,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 08:58:13,634 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-24 08:58:13,634 WARN [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-24 08:58:13,635 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 08:58:13,637 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-24 08:58:13,639 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-24 08:58:13,740 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/hbase.id with ID: f9fd94f9-4293-4895-b84c-8b66a6c97df1 2023-04-24 08:58:13,785 INFO [master/jenkins-hbase16:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-24 08:58:13,800 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 08:58:13,839 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x74692548 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:13,874 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210f3ae0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:13,899 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 08:58:13,901 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-24 08:58:13,918 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-24 08:58:13,918 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-24 08:58:13,920 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:13,923 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:13,925 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-24 08:58:13,963 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store-tmp 2023-04-24 08:58:14,390 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:14,390 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-24 08:58:14,391 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 08:58:14,391 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 08:58:14,391 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-24 08:58:14,391 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 08:58:14,391 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 08:58:14,391 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-24 08:58:14,393 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/WALs/jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 08:58:14,411 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C40429%2C1682326692433, suffix=, logDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/WALs/jenkins-hbase16.apache.org,40429,1682326692433, archiveDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/oldWALs, maxLogs=10 2023-04-24 08:58:14,466 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37329,DS-65d8d798-0a6a-484f-9e53-119be27d2c27,DISK] 2023-04-24 08:58:14,472 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:14,524 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/WALs/jenkins-hbase16.apache.org,40429,1682326692433/jenkins-hbase16.apache.org%2C40429%2C1682326692433.1682326694421 2023-04-24 08:58:14,525 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37329,DS-65d8d798-0a6a-484f-9e53-119be27d2c27,DISK]] 2023-04-24 08:58:14,525 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:58:14,526 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:14,529 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-24 08:58:14,530 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-24 08:58:14,583 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-24 08:58:14,591 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-24 08:58:14,610 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-24 08:58:14,622 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:14,628 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-24 08:58:14,629 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-24 08:58:14,646 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-24 08:58:14,650 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 08:58:14,651 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69683329, jitterRate=0.038362517952919006}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-24 08:58:14,651 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-24 08:58:14,652 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-24 08:58:14,670 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-24 08:58:14,671 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-24 08:58:14,672 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-24 08:58:14,674 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-24 08:58:14,699 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 24 msec 2023-04-24 08:58:14,699 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-24 08:58:14,722 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-24 08:58:14,727 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-24 08:58:14,746 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-24 08:58:14,749 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-24 08:58:14,751 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-24 08:58:14,755 INFO [master/jenkins-hbase16:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-24 08:58:14,759 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-24 08:58:14,792 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 08:58:14,794 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-24 08:58:14,795 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-24 08:58:14,808 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-24 08:58:14,817 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-24 08:58:14,817 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-24 08:58:14,817 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 08:58:14,817 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase16.apache.org,40429,1682326692433, sessionid=0x101ba998db50000, setting cluster-up flag (Was=false) 2023-04-24 08:58:14,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 08:58:14,867 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-24 08:58:14,869 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 08:58:14,883 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 08:58:14,908 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-24 08:58:14,910 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 08:58:14,912 WARN [master/jenkins-hbase16:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.hbase-snapshot/.tmp 2023-04-24 08:58:14,987 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(952): ClusterId : f9fd94f9-4293-4895-b84c-8b66a6c97df1 2023-04-24 08:58:14,991 DEBUG [RS:0;jenkins-hbase16:42837] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-24 08:58:15,002 DEBUG [RS:0;jenkins-hbase16:42837] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-24 08:58:15,002 DEBUG [RS:0;jenkins-hbase16:42837] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-24 08:58:15,007 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-24 08:58:15,010 DEBUG [RS:0;jenkins-hbase16:42837] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-24 08:58:15,010 DEBUG [RS:0;jenkins-hbase16:42837] zookeeper.ReadOnlyZKClient(139): Connect 0x0c53079d to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:15,018 DEBUG [RS:0;jenkins-hbase16:42837] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@294f867, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:15,018 DEBUG [RS:0;jenkins-hbase16:42837] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ab6ced, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-04-24 08:58:15,019 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-04-24 08:58:15,019 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-04-24 08:58:15,019 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-04-24 08:58:15,019 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-04-24 08:58:15,020 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase16:0, corePoolSize=10, maxPoolSize=10 2023-04-24 08:58:15,020 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,020 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-04-24 08:58:15,020 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,024 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682326725024 2023-04-24 08:58:15,026 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-24 08:58:15,028 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-24 08:58:15,029 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-24 08:58:15,034 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-24 08:58:15,037 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-24 08:58:15,039 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase16:42837 2023-04-24 08:58:15,043 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-24 08:58:15,043 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-24 08:58:15,043 INFO [RS:0;jenkins-hbase16:42837] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-24 08:58:15,043 INFO [RS:0;jenkins-hbase16:42837] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-24 08:58:15,043 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1023): About to register with Master. 2023-04-24 08:58:15,043 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-24 08:58:15,043 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-24 08:58:15,044 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,045 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-24 08:58:15,046 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase16.apache.org,40429,1682326692433 with isa=jenkins-hbase16.apache.org/188.40.62.62:42837, startcode=1682326693550 2023-04-24 08:58:15,047 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-24 08:58:15,048 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-24 08:58:15,049 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-24 08:58:15,050 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-24 08:58:15,051 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1682326695051,5,FailOnTimeoutGroup] 2023-04-24 08:58:15,051 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1682326695051,5,FailOnTimeoutGroup] 2023-04-24 08:58:15,051 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,052 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-24 08:58:15,053 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,053 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,064 DEBUG [RS:0;jenkins-hbase16:42837] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-24 08:58:15,105 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:49921, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-24 08:58:15,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:15,126 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f 2023-04-24 08:58:15,126 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:46389 2023-04-24 08:58:15,126 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-24 08:58:15,133 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-24 08:58:15,134 DEBUG [RS:0;jenkins-hbase16:42837] zookeeper.ZKUtil(162): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:15,134 WARN [RS:0;jenkins-hbase16:42837] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-24 08:58:15,135 INFO [RS:0;jenkins-hbase16:42837] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-24 08:58:15,135 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/WALs/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:15,136 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,42837,1682326693550] 2023-04-24 08:58:15,144 DEBUG [RS:0;jenkins-hbase16:42837] zookeeper.ZKUtil(162): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:15,154 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-24 08:58:15,161 INFO [RS:0;jenkins-hbase16:42837] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-24 08:58:15,182 INFO [RS:0;jenkins-hbase16:42837] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-24 08:58:15,186 INFO [RS:0;jenkins-hbase16:42837] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-24 08:58:15,186 INFO [RS:0;jenkins-hbase16:42837] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,187 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-24 08:58:15,194 INFO [RS:0;jenkins-hbase16:42837] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,195 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,195 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,195 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,195 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,196 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,196 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-04-24 08:58:15,196 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,196 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,196 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,196 DEBUG [RS:0;jenkins-hbase16:42837] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-04-24 08:58:15,198 INFO [RS:0;jenkins-hbase16:42837] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,198 INFO [RS:0;jenkins-hbase16:42837] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,198 INFO [RS:0;jenkins-hbase16:42837] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,214 INFO [RS:0;jenkins-hbase16:42837] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-24 08:58:15,216 INFO [RS:0;jenkins-hbase16:42837] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,42837,1682326693550-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:15,230 INFO [RS:0;jenkins-hbase16:42837] regionserver.Replication(203): jenkins-hbase16.apache.org,42837,1682326693550 started 2023-04-24 08:58:15,230 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1638): Serving as jenkins-hbase16.apache.org,42837,1682326693550, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:42837, sessionid=0x101ba998db50001 2023-04-24 08:58:15,230 DEBUG [RS:0;jenkins-hbase16:42837] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-24 08:58:15,230 DEBUG [RS:0;jenkins-hbase16:42837] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:15,230 DEBUG [RS:0;jenkins-hbase16:42837] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,42837,1682326693550' 2023-04-24 08:58:15,230 DEBUG [RS:0;jenkins-hbase16:42837] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:15,231 DEBUG [RS:0;jenkins-hbase16:42837] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:15,231 DEBUG [RS:0;jenkins-hbase16:42837] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-24 08:58:15,231 DEBUG [RS:0;jenkins-hbase16:42837] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-24 08:58:15,232 DEBUG [RS:0;jenkins-hbase16:42837] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:15,232 DEBUG [RS:0;jenkins-hbase16:42837] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,42837,1682326693550' 2023-04-24 08:58:15,232 DEBUG [RS:0;jenkins-hbase16:42837] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-24 08:58:15,232 DEBUG [RS:0;jenkins-hbase16:42837] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-24 08:58:15,232 DEBUG [RS:0;jenkins-hbase16:42837] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-24 08:58:15,232 INFO [RS:0;jenkins-hbase16:42837] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-24 08:58:15,233 INFO [RS:0;jenkins-hbase16:42837] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-24 08:58:15,349 INFO [RS:0;jenkins-hbase16:42837] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C42837%2C1682326693550, suffix=, logDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/WALs/jenkins-hbase16.apache.org,42837,1682326693550, archiveDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/oldWALs, maxLogs=32 2023-04-24 08:58:15,365 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37329,DS-65d8d798-0a6a-484f-9e53-119be27d2c27,DISK] 2023-04-24 08:58:15,368 INFO [RS:0;jenkins-hbase16:42837] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/WALs/jenkins-hbase16.apache.org,42837,1682326693550/jenkins-hbase16.apache.org%2C42837%2C1682326693550.1682326695352 2023-04-24 08:58:15,368 DEBUG [RS:0;jenkins-hbase16:42837] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37329,DS-65d8d798-0a6a-484f-9e53-119be27d2c27,DISK]] 2023-04-24 08:58:15,466 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-24 08:58:15,469 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-24 08:58:15,469 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f 2023-04-24 08:58:15,483 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:15,486 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-24 08:58:15,489 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/info 2023-04-24 08:58:15,490 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-24 08:58:15,491 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:15,491 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-24 08:58:15,494 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/rep_barrier 2023-04-24 08:58:15,494 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-24 08:58:15,495 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:15,495 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-24 08:58:15,497 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/table 2023-04-24 08:58:15,498 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-24 08:58:15,499 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:15,500 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740 2023-04-24 08:58:15,501 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740 2023-04-24 08:58:15,504 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 08:58:15,506 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-24 08:58:15,509 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 08:58:15,510 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64928788, jitterRate=-0.03248566389083862}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 08:58:15,510 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-24 08:58:15,510 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-24 08:58:15,510 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-24 08:58:15,510 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-24 08:58:15,511 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-24 08:58:15,511 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-24 08:58:15,512 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-24 08:58:15,512 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-24 08:58:15,517 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-24 08:58:15,517 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-24 08:58:15,524 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-24 08:58:15,536 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-24 08:58:15,538 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-24 08:58:15,693 DEBUG [jenkins-hbase16:40429] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-24 08:58:15,701 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,42837,1682326693550, state=OPENING 2023-04-24 08:58:15,742 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-24 08:58:15,750 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 08:58:15,751 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-24 08:58:15,756 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:58:15,945 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:15,946 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-24 08:58:15,950 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-24 08:58:15,960 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-24 08:58:15,960 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-24 08:58:15,964 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C42837%2C1682326693550.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/WALs/jenkins-hbase16.apache.org,42837,1682326693550, archiveDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/oldWALs, maxLogs=32 2023-04-24 08:58:15,980 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37329,DS-65d8d798-0a6a-484f-9e53-119be27d2c27,DISK] 2023-04-24 08:58:15,984 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/WALs/jenkins-hbase16.apache.org,42837,1682326693550/jenkins-hbase16.apache.org%2C42837%2C1682326693550.meta.1682326695966.meta 2023-04-24 08:58:15,984 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37329,DS-65d8d798-0a6a-484f-9e53-119be27d2c27,DISK]] 2023-04-24 08:58:15,985 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:58:15,987 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-24 08:58:16,003 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-24 08:58:16,009 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-24 08:58:16,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-24 08:58:16,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:16,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-24 08:58:16,013 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-24 08:58:16,016 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-24 08:58:16,018 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/info 2023-04-24 08:58:16,018 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/info 2023-04-24 08:58:16,019 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-24 08:58:16,019 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:16,019 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-24 08:58:16,021 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/rep_barrier 2023-04-24 08:58:16,021 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/rep_barrier 2023-04-24 08:58:16,021 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-24 08:58:16,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:16,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-24 08:58:16,024 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/table 2023-04-24 08:58:16,024 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/table 2023-04-24 08:58:16,024 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-24 08:58:16,025 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:16,027 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740 2023-04-24 08:58:16,029 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740 2023-04-24 08:58:16,032 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 08:58:16,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-24 08:58:16,036 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66651078, jitterRate=-0.006821542978286743}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 08:58:16,036 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-24 08:58:16,043 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682326695939 2023-04-24 08:58:16,057 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-24 08:58:16,058 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-24 08:58:16,058 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,42837,1682326693550, state=OPEN 2023-04-24 08:58:16,066 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-24 08:58:16,067 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-24 08:58:16,072 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-24 08:58:16,073 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,42837,1682326693550 in 311 msec 2023-04-24 08:58:16,080 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-24 08:58:16,080 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 550 msec 2023-04-24 08:58:16,086 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1320 sec 2023-04-24 08:58:16,086 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682326696086, completionTime=-1 2023-04-24 08:58:16,086 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-24 08:58:16,087 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-24 08:58:16,151 DEBUG [hconnection-0x7635eff-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:16,160 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57304, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:16,178 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-24 08:58:16,178 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682326756178 2023-04-24 08:58:16,178 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682326816178 2023-04-24 08:58:16,179 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 91 msec 2023-04-24 08:58:16,211 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40429,1682326692433-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:16,212 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40429,1682326692433-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:16,212 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40429,1682326692433-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:16,214 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase16:40429, period=300000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:16,214 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-24 08:58:16,221 DEBUG [master/jenkins-hbase16:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-24 08:58:16,231 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-24 08:58:16,232 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-24 08:58:16,241 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-24 08:58:16,244 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 08:58:16,246 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 08:58:16,265 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:16,268 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1 empty. 2023-04-24 08:58:16,269 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:16,269 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-24 08:58:16,695 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-24 08:58:16,699 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 81147daa64e17749d4bc895e5511dda1, NAME => 'hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp 2023-04-24 08:58:17,116 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:17,117 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 81147daa64e17749d4bc895e5511dda1, disabling compactions & flushes 2023-04-24 08:58:17,117 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 08:58:17,117 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 08:58:17,117 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. after waiting 0 ms 2023-04-24 08:58:17,117 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 08:58:17,117 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 08:58:17,117 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 81147daa64e17749d4bc895e5511dda1: 2023-04-24 08:58:17,122 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 08:58:17,137 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682326697124"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326697124"}]},"ts":"1682326697124"} 2023-04-24 08:58:17,158 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 08:58:17,160 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 08:58:17,164 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326697160"}]},"ts":"1682326697160"} 2023-04-24 08:58:17,168 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-24 08:58:17,204 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=81147daa64e17749d4bc895e5511dda1, ASSIGN}] 2023-04-24 08:58:17,209 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=81147daa64e17749d4bc895e5511dda1, ASSIGN 2023-04-24 08:58:17,212 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=81147daa64e17749d4bc895e5511dda1, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=false 2023-04-24 08:58:17,363 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=81147daa64e17749d4bc895e5511dda1, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:17,364 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682326697363"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326697363"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326697363"}]},"ts":"1682326697363"} 2023-04-24 08:58:17,373 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 81147daa64e17749d4bc895e5511dda1, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:58:17,536 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 08:58:17,537 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 81147daa64e17749d4bc895e5511dda1, NAME => 'hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1.', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:58:17,538 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:17,538 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:17,538 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:17,539 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:17,541 INFO [StoreOpener-81147daa64e17749d4bc895e5511dda1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:17,544 DEBUG [StoreOpener-81147daa64e17749d4bc895e5511dda1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1/info 2023-04-24 08:58:17,544 DEBUG [StoreOpener-81147daa64e17749d4bc895e5511dda1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1/info 2023-04-24 08:58:17,545 INFO [StoreOpener-81147daa64e17749d4bc895e5511dda1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81147daa64e17749d4bc895e5511dda1 columnFamilyName info 2023-04-24 08:58:17,546 INFO [StoreOpener-81147daa64e17749d4bc895e5511dda1-1] regionserver.HStore(310): Store=81147daa64e17749d4bc895e5511dda1/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:17,549 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:17,550 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:17,557 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 81147daa64e17749d4bc895e5511dda1 2023-04-24 08:58:17,561 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 08:58:17,562 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 81147daa64e17749d4bc895e5511dda1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64998274, jitterRate=-0.031450241804122925}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-24 08:58:17,562 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 81147daa64e17749d4bc895e5511dda1: 2023-04-24 08:58:17,564 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1., pid=6, masterSystemTime=1682326697529 2023-04-24 08:58:17,567 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 08:58:17,567 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 08:58:17,569 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=81147daa64e17749d4bc895e5511dda1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:17,569 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682326697568"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326697568"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326697568"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326697568"}]},"ts":"1682326697568"} 2023-04-24 08:58:17,576 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-24 08:58:17,576 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 81147daa64e17749d4bc895e5511dda1, server=jenkins-hbase16.apache.org,42837,1682326693550 in 200 msec 2023-04-24 08:58:17,579 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-24 08:58:17,580 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=81147daa64e17749d4bc895e5511dda1, ASSIGN in 372 msec 2023-04-24 08:58:17,581 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 08:58:17,582 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326697582"}]},"ts":"1682326697582"} 2023-04-24 08:58:17,585 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-24 08:58:17,594 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 08:58:17,597 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3610 sec 2023-04-24 08:58:17,646 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-24 08:58:17,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-24 08:58:17,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 08:58:17,703 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-24 08:58:17,725 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-24 08:58:17,739 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 44 msec 2023-04-24 08:58:17,747 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-24 08:58:17,766 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-24 08:58:17,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 32 msec 2023-04-24 08:58:17,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-24 08:58:17,825 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-24 08:58:17,826 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.191sec 2023-04-24 08:58:17,829 INFO [master/jenkins-hbase16:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-24 08:58:17,831 INFO [master/jenkins-hbase16:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-24 08:58:17,831 INFO [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-24 08:58:17,833 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40429,1682326692433-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-24 08:58:17,834 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40429,1682326692433-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-24 08:58:17,844 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-24 08:58:17,899 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x7256a797 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:17,909 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c146c14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:17,927 DEBUG [hconnection-0x4e832792-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:17,938 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:17,949 INFO [Listener at localhost.localdomain/44291] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 08:58:17,967 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=225, ProcessCount=169, AvailableMemoryMB=9127 2023-04-24 08:58:17,986 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 08:58:17,990 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 08:58:17,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 08:58:18,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 08:58:18,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:18,007 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 08:58:18,009 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 08:58:18,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-24 08:58:18,013 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,014 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe empty. 2023-04-24 08:58:18,016 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,017 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 08:58:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 08:58:18,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 08:58:18,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 08:58:18,436 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 08:58:18,440 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a1f74229b819199749846ce3feed93fe, NAME => 'TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp 2023-04-24 08:58:18,453 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:18,453 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a1f74229b819199749846ce3feed93fe, disabling compactions & flushes 2023-04-24 08:58:18,453 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:18,453 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:18,453 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. after waiting 0 ms 2023-04-24 08:58:18,453 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:18,453 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:18,453 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:18,458 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 08:58:18,460 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326698459"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326698459"}]},"ts":"1682326698459"} 2023-04-24 08:58:18,463 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 08:58:18,464 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 08:58:18,464 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326698464"}]},"ts":"1682326698464"} 2023-04-24 08:58:18,467 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 08:58:18,509 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a1f74229b819199749846ce3feed93fe, ASSIGN}] 2023-04-24 08:58:18,514 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a1f74229b819199749846ce3feed93fe, ASSIGN 2023-04-24 08:58:18,517 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a1f74229b819199749846ce3feed93fe, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=false 2023-04-24 08:58:18,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 08:58:18,669 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a1f74229b819199749846ce3feed93fe, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:18,670 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326698669"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326698669"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326698669"}]},"ts":"1682326698669"} 2023-04-24 08:58:18,673 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:58:18,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:18,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a1f74229b819199749846ce3feed93fe, NAME => 'TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:58:18,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:18,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,838 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,840 DEBUG [StoreOpener-a1f74229b819199749846ce3feed93fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A 2023-04-24 08:58:18,840 DEBUG [StoreOpener-a1f74229b819199749846ce3feed93fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A 2023-04-24 08:58:18,841 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:18,842 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a1f74229b819199749846ce3feed93fe columnFamilyName A 2023-04-24 08:58:18,843 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.HStore(310): Store=a1f74229b819199749846ce3feed93fe/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:18,843 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,845 DEBUG [StoreOpener-a1f74229b819199749846ce3feed93fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B 2023-04-24 08:58:18,845 DEBUG [StoreOpener-a1f74229b819199749846ce3feed93fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B 2023-04-24 08:58:18,845 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:18,846 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a1f74229b819199749846ce3feed93fe columnFamilyName B 2023-04-24 08:58:18,847 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.HStore(310): Store=a1f74229b819199749846ce3feed93fe/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:18,847 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,849 DEBUG [StoreOpener-a1f74229b819199749846ce3feed93fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C 2023-04-24 08:58:18,849 DEBUG [StoreOpener-a1f74229b819199749846ce3feed93fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C 2023-04-24 08:58:18,850 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:18,850 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a1f74229b819199749846ce3feed93fe columnFamilyName C 2023-04-24 08:58:18,851 INFO [StoreOpener-a1f74229b819199749846ce3feed93fe-1] regionserver.HStore(310): Store=a1f74229b819199749846ce3feed93fe/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:18,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:18,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 08:58:18,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:18,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 08:58:18,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened a1f74229b819199749846ce3feed93fe; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64559152, jitterRate=-0.037993669509887695}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 08:58:18,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:18,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., pid=11, masterSystemTime=1682326698827 2023-04-24 08:58:18,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:18,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:18,867 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a1f74229b819199749846ce3feed93fe, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:18,868 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326698867"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326698867"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326698867"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326698867"}]},"ts":"1682326698867"} 2023-04-24 08:58:18,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-24 08:58:18,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 in 197 msec 2023-04-24 08:58:18,879 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-24 08:58:18,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a1f74229b819199749846ce3feed93fe, ASSIGN in 365 msec 2023-04-24 08:58:18,881 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 08:58:18,881 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326698881"}]},"ts":"1682326698881"} 2023-04-24 08:58:18,884 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 08:58:18,919 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 08:58:18,922 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 917 msec 2023-04-24 08:58:19,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-24 08:58:19,142 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-24 08:58:19,151 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x482e2663 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,159 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@621130e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,162 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,166 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57320, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,169 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 08:58:19,172 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 08:58:19,179 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x5d114eb9 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,192 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@782365ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,192 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x6194be61 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,201 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@554a753e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,201 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x04d5a66f to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,209 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8fdb2b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,210 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x7de00027 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,218 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31222f6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,219 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x0446e24c to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,226 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29fd8399, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,227 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x7abbc496 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,242 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7386ab2c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,242 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x13065328 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,277 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@373daaaa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,278 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x11a028ee to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,293 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b12b88a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,294 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x5a6a210a to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:19,310 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a9d95f1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:19,320 DEBUG [hconnection-0x1403583-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,320 DEBUG [hconnection-0x1023f0e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,320 DEBUG [hconnection-0xed24095-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,321 DEBUG [hconnection-0x7eda9e41-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,321 DEBUG [hconnection-0x1220839d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,322 DEBUG [hconnection-0x39c3569-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,323 DEBUG [hconnection-0x7cb4f2d6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,324 DEBUG [hconnection-0xd204e20-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,324 DEBUG [hconnection-0x4ac352ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:19,324 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,351 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,352 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,356 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,356 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57356, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,357 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:19,363 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,371 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,373 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57394, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:19,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:19,387 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:19,387 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:19,387 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:19,387 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:19,388 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:19,388 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:19,391 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:19,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:19,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:19,393 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:19,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:19,395 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,395 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:19,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:19,400 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:19,401 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:19,402 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:19,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:19,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:19,421 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:19,422 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:19,422 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:58:19,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:19,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:19,434 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:19,435 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:19,436 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:19,436 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:19,436 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:19,436 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:19,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326699492, max timestamp=1682326699510Num uniques 859; 2023-04-24 08:58:19,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:19,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326699492, max timestamp=1682326699510Num uniques 859; 2023-04-24 08:58:19,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:19,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326699492, max timestamp=1682326699510Num uniques 859; 2023-04-24 08:58:19,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:19,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326759564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326759582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326759587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326759582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326759587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326759715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326759714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326759715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326759716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326759716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326759925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326759930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326759929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326759930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326759932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:19,979 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/6987e13f212045d897e3a9a06fa084fc 2023-04-24 08:58:20,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326760234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326760236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326760237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326760237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326760237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,471 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/4fdb3c0b5fbe44acb73fe727073003f1 2023-04-24 08:58:20,520 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/4ea874bce4a34867b606c7c56bf7c32c 2023-04-24 08:58:20,535 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/6987e13f212045d897e3a9a06fa084fc as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/6987e13f212045d897e3a9a06fa084fc 2023-04-24 08:58:20,549 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/6987e13f212045d897e3a9a06fa084fc, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 08:58:20,556 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/4fdb3c0b5fbe44acb73fe727073003f1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4fdb3c0b5fbe44acb73fe727073003f1 2023-04-24 08:58:20,575 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4fdb3c0b5fbe44acb73fe727073003f1, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 08:58:20,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/4ea874bce4a34867b606c7c56bf7c32c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4ea874bce4a34867b606c7c56bf7c32c 2023-04-24 08:58:20,595 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4ea874bce4a34867b606c7c56bf7c32c, entries=150, sequenceid=12, filesize=11.6 K 2023-04-24 08:58:20,602 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a1f74229b819199749846ce3feed93fe in 1179ms, sequenceid=12, compaction requested=false 2023-04-24 08:58:20,604 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-24 08:58:20,607 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:20,607 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:20,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:20,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:20,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:20,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:20,650 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,650 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:20,650 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,650 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:20,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:20,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47b9bd49[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:20,653 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:20,653 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:20,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:20,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:20,659 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,659 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:20,668 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:20,668 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:20,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:20,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:20,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:20,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:20,675 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:20,675 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:20,675 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:20,675 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:20,675 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:20,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:20,685 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:20,688 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,709 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,709 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:20,709 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:20,709 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:20,709 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:20,709 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,710 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:20,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:20,710 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:20,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:20,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,715 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:20,715 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:20,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:20,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:20,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:20,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-24 08:58:20,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:20,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:20,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:20,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:20,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:20,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:20,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326760762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326760763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326760769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326760771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326760773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,815 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:20,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:20,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:20,832 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:20,832 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:20,833 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:20,833 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:20,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:20,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:20,841 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:20,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:20,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:20,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:20,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:20,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:20,844 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:20,844 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:20,845 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:20,845 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:20,845 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:20,845 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:20,845 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:20,846 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:20,846 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:20,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:20,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:20,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:20,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:20,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:20,860 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,860 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:20,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:20,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6160bde0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:20,862 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:20,863 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,866 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,867 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:20,867 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:20,867 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:20,868 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,868 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:20,876 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,876 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:20,876 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:20,876 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326760882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326760882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:20,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:20,887 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:20,887 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:20,887 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:20,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,887 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:20,887 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:20,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326760884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326760884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:20,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326760886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,918 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,918 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:20,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,918 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:20,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:20,918 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,921 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,942 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:20,942 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:20,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:20,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:20,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:20,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:20,942 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,942 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:20,942 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:20,942 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:20,943 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:20,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:20,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:20,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:20,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:21,048 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:21,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:21,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:21,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:21,059 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:21,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:21,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:21,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:21,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,067 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:21,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:21,068 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,068 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:21,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:21,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:21,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:21,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:21,094 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:21,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:21,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:21,096 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:21,096 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:21,096 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:21,096 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:21,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326761097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,108 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:21,111 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:21,112 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,112 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:21,112 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:21,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326761097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326761097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326761097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326761097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,125 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:21,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fbcf73[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:21,141 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:21,141 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,150 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:21,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:21,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:21,150 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,151 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:21,159 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:21,159 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:21,159 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:21,160 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:21,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:21,165 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:21,165 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:21,165 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:21,165 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:21,165 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:21,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:21,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,177 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,179 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,200 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,200 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,200 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,200 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,200 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:21,200 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:21,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:21,200 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:21,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:21,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:21,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:21,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:21,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:21,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:21,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/55f28a36e7c24c9682f79c45721fdc4b 2023-04-24 08:58:21,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/c1dca11619574eb283333edad3a2ece1 2023-04-24 08:58:21,306 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:21,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:21,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:21,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:21,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:21,332 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:21,332 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:21,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:21,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:21,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326761411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326761421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326761421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326761422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326761424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:21,435 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:21,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:21,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:21,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:21,440 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:21,440 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:21,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:21,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:21,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:21,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:21,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:21,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:21,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:21,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:21,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:21,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:21,446 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:21,581 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:21,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5becc675[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:21,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:21,587 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,600 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,600 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:21,600 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:21,600 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:21,601 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,601 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:21,609 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,609 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,609 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:21,610 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:21,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:21,613 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:21,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:21,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:21,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:21,614 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:21,625 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,626 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,625 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,626 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:21,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,642 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 08:58:21,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:21,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,659 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,659 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,659 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,659 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,659 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,659 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:21,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:21,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:21,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:21,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:21,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:21,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:21,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:21,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:21,734 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-24 08:58:21,735 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-24 08:58:21,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:21,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:21,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:21,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:21,772 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:21,772 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:21,776 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:21,776 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:21,777 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,777 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,783 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:21,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:21,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:21,783 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,784 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:21,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:21,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:21,785 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:21,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:21,788 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:21,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:21,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:21,789 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:21,789 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:21,789 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:21,789 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:21,790 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:21,790 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:21,791 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,791 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:21,791 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:21,800 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,800 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,800 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,800 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:21,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b57bb4[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:21,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:21,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:21,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:21,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:21,809 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,809 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:21,816 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:21,816 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:21,816 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:21,817 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:21,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:21,826 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:21,826 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:21,826 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:21,826 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:21,826 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:21,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:21,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:21,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:21,835 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,838 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,869 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,869 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,869 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:21,869 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,869 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,869 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:21,869 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:21,876 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:21,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:21,869 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:21,877 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:21,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:21,877 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:21,877 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:21,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:21,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/8591c0afc367445eb27359876fd902ae 2023-04-24 08:58:21,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/55f28a36e7c24c9682f79c45721fdc4b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/55f28a36e7c24c9682f79c45721fdc4b 2023-04-24 08:58:21,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326761920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326761926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/55f28a36e7c24c9682f79c45721fdc4b, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 08:58:21,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326761926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/c1dca11619574eb283333edad3a2ece1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c1dca11619574eb283333edad3a2ece1 2023-04-24 08:58:21,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326761930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:21,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326761941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:21,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c1dca11619574eb283333edad3a2ece1, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 08:58:21,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/8591c0afc367445eb27359876fd902ae as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8591c0afc367445eb27359876fd902ae 2023-04-24 08:58:21,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8591c0afc367445eb27359876fd902ae, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 08:58:21,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for a1f74229b819199749846ce3feed93fe in 1214ms, sequenceid=48, compaction requested=false 2023-04-24 08:58:21,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:21,977 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:21,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:21,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:21,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:21,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:21,992 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:21,992 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:21,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:21,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:21,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:21,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:22,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:22,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:22,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:22,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:22,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:22,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:22,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:22,017 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:22,017 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:22,017 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:22,017 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:22,017 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:58:22,017 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:22,018 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:22,018 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:22,018 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:22,018 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:22,018 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:22,076 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/9020a8b57b3a4396941f0b95412410d8 2023-04-24 08:58:22,130 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/44d8002a1fff44e8b6dda2a6860c92e6 2023-04-24 08:58:22,236 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/f9704429a62e4c3381de8172c8d01a23 2023-04-24 08:58:22,255 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/9020a8b57b3a4396941f0b95412410d8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9020a8b57b3a4396941f0b95412410d8 2023-04-24 08:58:22,269 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9020a8b57b3a4396941f0b95412410d8, entries=150, sequenceid=58, filesize=11.6 K 2023-04-24 08:58:22,272 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/44d8002a1fff44e8b6dda2a6860c92e6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/44d8002a1fff44e8b6dda2a6860c92e6 2023-04-24 08:58:22,291 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/44d8002a1fff44e8b6dda2a6860c92e6, entries=150, sequenceid=58, filesize=11.6 K 2023-04-24 08:58:22,294 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/f9704429a62e4c3381de8172c8d01a23 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f9704429a62e4c3381de8172c8d01a23 2023-04-24 08:58:22,310 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f9704429a62e4c3381de8172c8d01a23, entries=150, sequenceid=58, filesize=11.6 K 2023-04-24 08:58:22,313 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for a1f74229b819199749846ce3feed93fe in 296ms, sequenceid=58, compaction requested=true 2023-04-24 08:58:22,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:22,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:22,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:22,314 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:22,314 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,314 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:22,314 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:22,377 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,377 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:22,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@355e53df[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:22,382 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:22,382 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,391 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,392 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:22,392 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:22,392 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:22,393 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,394 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:22,400 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:22,400 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:22,400 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:22,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:22,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:22,411 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:22,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:22,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:22,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:22,411 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:22,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,417 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,417 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:22,418 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,420 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,433 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,433 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,433 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:22,433 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,434 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:22,433 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:22,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:22,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:22,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:22,434 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:22,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,435 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:22,435 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:22,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:22,536 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:22,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:22,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:22,541 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:22,541 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:22,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:22,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:22,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,550 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:22,550 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:22,550 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,550 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:22,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:22,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:22,552 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:22,552 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:22,553 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:22,553 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:22,553 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:22,553 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:22,553 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:22,553 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:22,554 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:22,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:22,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:22,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:22,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:22,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,558 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,559 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,559 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:22,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd69250[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:22,561 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:22,562 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,568 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,568 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:22,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,568 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:22,568 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:22,568 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:22,575 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,575 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:22,575 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:22,575 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:22,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:22,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:22,583 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:22,583 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:22,583 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:22,583 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:22,583 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:22,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,592 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,593 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,595 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,609 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,609 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,609 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:22,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:22,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:22,609 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:22,609 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,610 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:22,610 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:22,610 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,610 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:22,610 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:22,610 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,610 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:22,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:22,614 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-24 08:58:22,615 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-24 08:58:22,710 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:22,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:22,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:22,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:22,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:22,718 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:22,718 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:22,720 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:22,720 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:22,721 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,721 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,725 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:22,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,726 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,726 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:22,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:22,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:22,727 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:22,727 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:22,730 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:22,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:22,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:22,744 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:22,744 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:22,744 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:22,744 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:22,744 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:22,744 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:22,744 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,744 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:22,744 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:22,750 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:22,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b434f1[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:22,752 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:22,752 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,758 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:22,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:22,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:22,758 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,758 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:22,766 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:22,766 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:22,767 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:22,767 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:22,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:22,772 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:22,772 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:22,772 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:22,772 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:22,772 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:22,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,785 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,787 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,820 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,820 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,820 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,820 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:22,820 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:22,821 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:22,820 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:22,821 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:22,821 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:22,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:22,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:22,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:22,829 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:22,829 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:22,929 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:22,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:22,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:22,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:22,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:22,936 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:22,936 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:22,936 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:22,937 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:22,937 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,937 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:22,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:58:22,947 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:22,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:22,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:22,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:22,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:22,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:22,950 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,950 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:22,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:22,950 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,950 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:22,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:22,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:22,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:22,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:22,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:22,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:22,962 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:22,962 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:22,967 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:22,967 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:22,967 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:22,967 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:22,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:22,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:22,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:22,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:22,976 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,976 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,976 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:22,976 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:22,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:22,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/51255a66b74d4479993999049494af02 2023-04-24 08:58:22,990 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:22,990 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:22,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aa68788[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:23,000 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,000 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:23,000 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:23,000 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:23,001 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,001 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:23,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,008 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:23,008 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,008 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:23,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:23,011 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:23,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:23,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:23,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:23,012 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:23,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,020 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,026 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,026 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,029 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:23,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:23,032 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,032 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,032 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,032 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,032 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,032 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:23,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:23,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/8261d3d6cc9d430bbb9fa2dc44bfd520 2023-04-24 08:58:23,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326702991, max timestamp=1682326703060Num uniques 859; 2023-04-24 08:58:23,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:23,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326702991, max timestamp=1682326703063Num uniques 859; 2023-04-24 08:58:23,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:23,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326702991, max timestamp=1682326703060Num uniques 859; 2023-04-24 08:58:23,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:23,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/255e8673985644a1a8eff570d6112075 2023-04-24 08:58:23,133 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:23,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326763125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326763126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,137 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/51255a66b74d4479993999049494af02 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/51255a66b74d4479993999049494af02 2023-04-24 08:58:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326763127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326763124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:23,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:23,146 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326763136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,146 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,147 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:23,147 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:23,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/51255a66b74d4479993999049494af02, entries=150, sequenceid=70, filesize=11.6 K 2023-04-24 08:58:23,160 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,160 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,161 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,161 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/8261d3d6cc9d430bbb9fa2dc44bfd520 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8261d3d6cc9d430bbb9fa2dc44bfd520 2023-04-24 08:58:23,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:23,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:23,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:23,178 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:23,178 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,186 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:23,186 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:23,186 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:23,186 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,186 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:23,186 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:23,186 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,186 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:23,186 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:23,187 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,187 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:23,187 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:23,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8261d3d6cc9d430bbb9fa2dc44bfd520, entries=100, sequenceid=70, filesize=9.3 K 2023-04-24 08:58:23,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/255e8673985644a1a8eff570d6112075 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/255e8673985644a1a8eff570d6112075 2023-04-24 08:58:23,200 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,200 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,200 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,200 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:23,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d21e3a8[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:23,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:23,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,208 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:23,208 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,208 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:23,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:23,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:23,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/255e8673985644a1a8eff570d6112075, entries=100, sequenceid=70, filesize=9.3 K 2023-04-24 08:58:23,212 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a1f74229b819199749846ce3feed93fe in 265ms, sequenceid=70, compaction requested=true 2023-04-24 08:58:23,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:23,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:23,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:23,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:23,215 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:23,215 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:23,217 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:23,217 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,217 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:23,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:23,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:23,224 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:23,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:23,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:23,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:23,224 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:23,228 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:23,230 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/B is initiating minor compaction (all files) 2023-04-24 08:58:23,230 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/B in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,230 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4fdb3c0b5fbe44acb73fe727073003f1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c1dca11619574eb283333edad3a2ece1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/44d8002a1fff44e8b6dda2a6860c92e6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8261d3d6cc9d430bbb9fa2dc44bfd520] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=44.1 K 2023-04-24 08:58:23,232 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4fdb3c0b5fbe44acb73fe727073003f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682326699405 2023-04-24 08:58:23,232 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47540 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:23,234 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/A is initiating minor compaction (all files) 2023-04-24 08:58:23,235 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,235 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,236 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,236 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/A in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,236 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/6987e13f212045d897e3a9a06fa084fc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/55f28a36e7c24c9682f79c45721fdc4b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9020a8b57b3a4396941f0b95412410d8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/51255a66b74d4479993999049494af02] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=46.4 K 2023-04-24 08:58:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,236 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682326703115, max timestamp=1682326703239Num uniques 1018; 2023-04-24 08:58:23,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 08:58:23,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682326703115, max timestamp=1682326703239Num uniques 1018; 2023-04-24 08:58:23,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 08:58:23,244 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c1dca11619574eb283333edad3a2ece1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682326699567 2023-04-24 08:58:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,245 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 44d8002a1fff44e8b6dda2a6860c92e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682326700761 2023-04-24 08:58:23,245 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 6987e13f212045d897e3a9a06fa084fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682326699405 2023-04-24 08:58:23,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=71.56 KB heapSize=198.72 KB 2023-04-24 08:58:23,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:23,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682326703115, max timestamp=1682326703239Num uniques 1018; 2023-04-24 08:58:23,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 08:58:23,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:23,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:23,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:23,246 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:23,246 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:23,247 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:23,248 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,248 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 8261d3d6cc9d430bbb9fa2dc44bfd520, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682326702941 2023-04-24 08:58:23,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,249 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 55f28a36e7c24c9682f79c45721fdc4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682326699567 2023-04-24 08:58:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,252 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 9020a8b57b3a4396941f0b95412410d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682326700761 2023-04-24 08:58:23,253 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 51255a66b74d4479993999049494af02, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682326702941 2023-04-24 08:58:23,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,262 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:23,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:23,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:23,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,263 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:23,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,263 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:23,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/2b67e509c97f425bbc0f21a0c2f30542 2023-04-24 08:58:23,306 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#A#compaction#14 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:23,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326703278, max timestamp=1682326703309Num uniques 859; 2023-04-24 08:58:23,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326703278, max timestamp=1682326703309Num uniques 859; 2023-04-24 08:58:23,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 08:58:23,314 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#B#compaction#13 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:23,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 08:58:23,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682326703278, max timestamp=1682326703309Num uniques 859; 2023-04-24 08:58:23,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-24 08:58:23,363 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:23,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:23,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:23,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:23,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:23,375 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:23,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,377 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:23,378 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:23,378 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,378 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,388 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,388 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,389 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,389 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:23,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:23,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:23,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:23,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:23,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,400 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:23,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682326703342, max timestamp=1682326703398Num uniques 1009; 2023-04-24 08:58:23,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=16848, suffix heap size=132904,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=368 2023-04-24 08:58:23,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:23,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:23,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:23,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:23,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:23,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:23,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:23,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:23,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682326703342, max timestamp=1682326703401Num uniques 1118; 2023-04-24 08:58:23,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=16848, suffix heap size=144240,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=368 2023-04-24 08:58:23,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1682326703342, max timestamp=1682326703398Num uniques 1118; 2023-04-24 08:58:23,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=16848, suffix heap size=144240,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=368 2023-04-24 08:58:23,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,409 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,409 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,409 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:23,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@346cb24e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:23,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:23,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,415 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/d57a3407fd3f4b96b6bde3d2bdd11d12 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/d57a3407fd3f4b96b6bde3d2bdd11d12 2023-04-24 08:58:23,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,418 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:23,418 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:23,418 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:23,420 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,420 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:23,425 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,425 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:23,425 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,425 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:23,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:23,446 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:23,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:23,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:23,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:23,447 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:23,453 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/B of a1f74229b819199749846ce3feed93fe into d57a3407fd3f4b96b6bde3d2bdd11d12(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:23,454 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:23,454 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/B, priority=12, startTime=1682326703215; duration=0sec 2023-04-24 08:58:23,455 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:23,455 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:23,458 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,458 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,458 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,460 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,465 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:23,465 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/C is initiating minor compaction (all files) 2023-04-24 08:58:23,465 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/C in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,465 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4ea874bce4a34867b606c7c56bf7c32c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8591c0afc367445eb27359876fd902ae, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f9704429a62e4c3381de8172c8d01a23, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/255e8673985644a1a8eff570d6112075] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=44.1 K 2023-04-24 08:58:23,466 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4ea874bce4a34867b606c7c56bf7c32c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682326699405 2023-04-24 08:58:23,467 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 8591c0afc367445eb27359876fd902ae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682326699567 2023-04-24 08:58:23,468 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f9704429a62e4c3381de8172c8d01a23, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682326700761 2023-04-24 08:58:23,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,476 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 255e8673985644a1a8eff570d6112075, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682326702941 2023-04-24 08:58:23,476 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:23,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:23,477 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,477 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,477 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326763454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,480 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:23,480 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:23,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:23,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326763454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326763455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326763475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326763477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,502 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#C#compaction#16 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:23,536 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/2a8f5501344243a39c2c45fea7aea6ff as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/2a8f5501344243a39c2c45fea7aea6ff 2023-04-24 08:58:23,551 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/C of a1f74229b819199749846ce3feed93fe into 2a8f5501344243a39c2c45fea7aea6ff(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:23,551 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:23,551 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/C, priority=12, startTime=1682326703215; duration=0sec 2023-04-24 08:58:23,551 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:23,580 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:23,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:23,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326763584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326763584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:23,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:23,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:23,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326763586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,589 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:23,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:23,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:23,590 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,590 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326763589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326763590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,635 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,635 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,636 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,636 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:23,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:23,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:23,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:23,637 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:23,637 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,638 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:23,638 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:23,638 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:23,638 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,638 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:23,638 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:23,638 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,638 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:23,638 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:23,638 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,638 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:23,638 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:23,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:23,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15de1b8e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:23,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:23,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,666 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,667 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:23,667 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:23,667 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:23,667 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,667 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:23,675 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:23,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:23,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:23,679 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:23,679 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:23,679 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:23,679 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:23,679 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:23,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,684 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,686 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:23,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:23,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,692 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:23,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,692 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:23,692 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,693 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,693 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,693 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,693 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:23,769 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/5969ff67bbd9421897da489a5e0c5270 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5969ff67bbd9421897da489a5e0c5270 2023-04-24 08:58:23,781 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/A of a1f74229b819199749846ce3feed93fe into 5969ff67bbd9421897da489a5e0c5270(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:23,782 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:23,782 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/A, priority=12, startTime=1682326703214; duration=0sec 2023-04-24 08:58:23,782 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:23,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326763787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326763787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326763792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,793 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:23,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:23,797 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:23,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,800 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:23,800 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:23,801 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,801 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:23,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326763795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326763794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/4c1d6acf22b04e43a87fe396cdbb69c7 2023-04-24 08:58:23,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,809 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,810 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,810 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:23,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:23,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:23,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:23,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:23,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,812 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:23,820 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:23,820 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,820 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:23,820 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:23,820 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:23,820 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:23,820 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:23,820 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:23,820 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,824 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:23,824 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:23,833 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,839 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,839 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:23,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0a547a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:23,851 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:23,851 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/1f5ff89df86741da8fa4e3435be71359 2023-04-24 08:58:23,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,858 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,858 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:23,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:23,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:23,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:23,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/2b67e509c97f425bbc0f21a0c2f30542 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/2b67e509c97f425bbc0f21a0c2f30542 2023-04-24 08:58:23,866 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:23,866 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,866 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,866 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:23,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:23,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:23,870 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:23,870 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:23,870 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:23,871 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:23,871 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:23,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:23,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,876 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:23,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:23,877 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/2b67e509c97f425bbc0f21a0c2f30542, entries=250, sequenceid=109, filesize=16.2 K 2023-04-24 08:58:23,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:23,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/4c1d6acf22b04e43a87fe396cdbb69c7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4c1d6acf22b04e43a87fe396cdbb69c7 2023-04-24 08:58:23,883 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,883 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:23,883 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:23,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:23,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:23,883 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:23,883 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:23,883 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:23,883 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:23,884 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:23,886 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:23,884 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:23,886 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:23,886 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:23,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:23,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4c1d6acf22b04e43a87fe396cdbb69c7, entries=150, sequenceid=109, filesize=11.6 K 2023-04-24 08:58:23,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/1f5ff89df86741da8fa4e3435be71359 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1f5ff89df86741da8fa4e3435be71359 2023-04-24 08:58:23,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1f5ff89df86741da8fa4e3435be71359, entries=150, sequenceid=109, filesize=11.6 K 2023-04-24 08:58:23,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=161.02 KB/164880 for a1f74229b819199749846ce3feed93fe in 684ms, sequenceid=109, compaction requested=false 2023-04-24 08:58:23,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:23,987 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:23,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:23,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:23,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:23,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:23,996 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:23,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:23,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:23,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:23,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:23,998 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:24,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:24,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:24,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:24,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:24,011 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:24,011 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:24,012 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:24,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:24,014 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:24,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:24,014 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:24,014 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=161.02 KB heapSize=445.08 KB 2023-04-24 08:58:24,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:24,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:24,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:24,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:24,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:24,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:24,062 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/d09093964f7041a78bbd0c7bba277461 2023-04-24 08:58:24,093 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/6fa81489bda647a888fda5ea72aefa03 2023-04-24 08:58:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:24,099 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:24,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326764105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326764107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326764107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326764111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326764114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326764216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326764219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326764220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326764419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326764422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326764423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,541 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/009863d17dfc4018a95a361606e63cce 2023-04-24 08:58:24,548 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/d09093964f7041a78bbd0c7bba277461 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/d09093964f7041a78bbd0c7bba277461 2023-04-24 08:58:24,555 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/d09093964f7041a78bbd0c7bba277461, entries=150, sequenceid=166, filesize=11.8 K 2023-04-24 08:58:24,558 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/6fa81489bda647a888fda5ea72aefa03 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6fa81489bda647a888fda5ea72aefa03 2023-04-24 08:58:24,564 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6fa81489bda647a888fda5ea72aefa03, entries=150, sequenceid=166, filesize=11.8 K 2023-04-24 08:58:24,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/009863d17dfc4018a95a361606e63cce as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/009863d17dfc4018a95a361606e63cce 2023-04-24 08:58:24,578 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/009863d17dfc4018a95a361606e63cce, entries=150, sequenceid=166, filesize=11.8 K 2023-04-24 08:58:24,580 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.03 KB/455712, currentSize=33.54 KB/34350 for a1f74229b819199749846ce3feed93fe in 566ms, sequenceid=166, compaction requested=true 2023-04-24 08:58:24,580 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:24,580 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:24,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:24,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:24,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:24,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:24,616 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,616 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:24,617 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,617 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:24,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:24,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6904d010[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:24,619 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:24,619 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:24,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:58:24,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:24,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:24,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:24,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:24,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:24,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:24,625 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,625 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,625 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,626 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:24,626 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:24,626 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:24,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,633 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:24,633 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,633 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:24,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:24,637 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:24,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:24,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:24,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:24,637 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:24,641 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,641 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,641 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:24,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:24,642 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:24,643 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,650 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:24,650 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:24,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:24,650 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,650 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,650 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,650 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,651 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,651 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,651 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,651 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:24,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:24,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:24,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:24,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:24,653 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:24,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:24,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/948dd94e78524c8da4d49f0b5841e01e 2023-04-24 08:58:24,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326704690, max timestamp=1682326704736Num uniques 859; 2023-04-24 08:58:24,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326704690, max timestamp=1682326704736Num uniques 859; 2023-04-24 08:58:24,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:24,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:24,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326704690, max timestamp=1682326704736Num uniques 859; 2023-04-24 08:58:24,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:24,753 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:24,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:24,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:24,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:24,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:24,758 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:24,758 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:24,759 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:24,759 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:24,760 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,760 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,766 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,766 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:24,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:24,767 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,767 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:24,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:24,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:24,769 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:24,769 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:24,770 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:24,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:24,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:24,771 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:24,771 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:24,771 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:24,771 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:24,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:24,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:24,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:24,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:24,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326764764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326764766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326704747, max timestamp=1682326704769Num uniques 1018; 2023-04-24 08:58:24,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 08:58:24,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326704747, max timestamp=1682326704769Num uniques 1018; 2023-04-24 08:58:24,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 08:58:24,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326704747, max timestamp=1682326704769Num uniques 1018; 2023-04-24 08:58:24,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 08:58:24,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,782 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:24,783 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,783 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:24,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:24,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38abf1bc[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:24,785 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:24,785 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,814 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:24,814 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:24,814 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,814 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:24,814 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:24,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326764817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326764817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,825 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:24,825 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:24,825 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:24,829 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326764822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:24,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:24,832 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:24,832 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:24,832 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:24,832 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:24,832 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:24,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:24,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:24,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:24,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,845 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,850 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:24,850 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:24,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:24,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:24,851 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,851 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,851 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,851 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:24,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:24,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:24,851 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,853 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:24,853 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:24,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:24,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326764879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326764879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326764924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326764924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:24,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326764932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,953 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:24,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:24,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:24,958 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:24,958 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:24,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:24,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:24,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,966 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,966 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:24,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:24,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:24,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:24,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:24,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:24,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:24,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:24,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:24,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:24,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:24,969 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:24,969 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:24,970 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:24,970 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:24,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:24,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:24,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:24,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:24,975 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,975 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:24,975 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,975 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:24,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:24,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcc8f4c[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:24,977 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:24,977 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,983 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,983 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,983 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:24,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:24,984 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:24,984 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:24,984 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:24,991 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,991 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:24,991 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:24,992 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:24,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:24,995 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:24,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:24,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:24,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:24,995 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:25,007 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,007 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,007 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:25,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,016 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:25,016 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:25,016 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:25,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:25,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:25,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:25,016 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:25,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:25,018 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:25,016 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:25,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326765083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326765083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,118 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:25,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:25,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:25,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:25,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:25,121 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:25,121 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:25,122 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:25,122 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:25,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/38f33704f6484b6381ae17dfcbbb7529 2023-04-24 08:58:25,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326765129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326765129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,133 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:25,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:25,133 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,134 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,134 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:25,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:25,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:25,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:25,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:25,143 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:25,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:25,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:25,144 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,144 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:25,144 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:25,144 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:25,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:25,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:25,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:25,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326765144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,152 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:25,160 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:25,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ecb90b[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:25,160 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,166 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,167 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:25,167 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:25,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:25,167 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:25,175 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:25,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,175 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:25,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,175 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:25,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:25,178 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:25,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:25,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:25,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:25,178 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:25,183 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,183 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,183 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:25,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:25,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,186 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,191 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:25,191 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:25,191 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,192 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:25,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:25,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:25,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:25,192 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,192 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:25,193 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,193 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,193 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:25,193 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,193 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:25,193 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:25,293 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:25,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:25,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:25,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:25,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:25,297 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:25,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:25,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:25,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:25,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,308 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:25,308 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:25,308 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,308 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:25,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:25,312 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:25,312 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:25,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:25,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:25,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:25,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:25,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:25,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:25,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:25,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:25,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:25,324 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,324 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,324 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,324 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:25,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ba767d6[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:25,326 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:25,326 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:25,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:25,333 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,334 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:25,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:25,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:25,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:25,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:25,365 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:25,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:25,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:25,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:25,365 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:25,375 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,375 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,375 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,377 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:25,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:25,383 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:25,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:25,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:25,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,384 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,385 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:25,385 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:25,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:25,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:25,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:25,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326765387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326765389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326765435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326765436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326765455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,485 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:25,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:25,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:25,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:25,488 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:25,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:25,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:25,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:25,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,500 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:25,500 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:25,500 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,500 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:25,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:25,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:25,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:25,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:25,504 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:25,504 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:25,504 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:25,504 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,504 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:25,504 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:25,505 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,505 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:25,505 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:25,505 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,505 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:25,505 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:25,516 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,517 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,517 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:25,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c66430f[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:25,519 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:25,519 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,525 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,525 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,525 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:25,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:25,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:25,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:25,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:25,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:25,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:25,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:25,537 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:25,537 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:25,537 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:25,537 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:25,537 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:25,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:25,542 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,542 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:25,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,543 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:25,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,550 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:25,550 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:25,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:25,550 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,550 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:25,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:25,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:25,550 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:25,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:25,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:25,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:25,551 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:25,551 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:25,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/3c90d70e96bd4a9ab83ade5f8c35cf8f 2023-04-24 08:58:25,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/948dd94e78524c8da4d49f0b5841e01e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/948dd94e78524c8da4d49f0b5841e01e 2023-04-24 08:58:25,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/948dd94e78524c8da4d49f0b5841e01e, entries=150, sequenceid=176, filesize=11.8 K 2023-04-24 08:58:25,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/38f33704f6484b6381ae17dfcbbb7529 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/38f33704f6484b6381ae17dfcbbb7529 2023-04-24 08:58:25,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/38f33704f6484b6381ae17dfcbbb7529, entries=150, sequenceid=176, filesize=11.8 K 2023-04-24 08:58:25,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/3c90d70e96bd4a9ab83ade5f8c35cf8f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3c90d70e96bd4a9ab83ade5f8c35cf8f 2023-04-24 08:58:25,593 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3c90d70e96bd4a9ab83ade5f8c35cf8f, entries=150, sequenceid=176, filesize=11.8 K 2023-04-24 08:58:25,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a1f74229b819199749846ce3feed93fe in 973ms, sequenceid=176, compaction requested=true 2023-04-24 08:58:25,595 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:25,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:25,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:25,595 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:25,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:25,595 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:25,600 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:25,600 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52662 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:25,600 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/B is initiating minor compaction (all files) 2023-04-24 08:58:25,600 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/A is initiating minor compaction (all files) 2023-04-24 08:58:25,600 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/B in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,600 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/A in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,600 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5969ff67bbd9421897da489a5e0c5270, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/2b67e509c97f425bbc0f21a0c2f30542, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/d09093964f7041a78bbd0c7bba277461, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/948dd94e78524c8da4d49f0b5841e01e] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=51.4 K 2023-04-24 08:58:25,600 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/d57a3407fd3f4b96b6bde3d2bdd11d12, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4c1d6acf22b04e43a87fe396cdbb69c7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6fa81489bda647a888fda5ea72aefa03, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/38f33704f6484b6381ae17dfcbbb7529] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=46.9 K 2023-04-24 08:58:25,601 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 5969ff67bbd9421897da489a5e0c5270, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682326700761 2023-04-24 08:58:25,602 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d57a3407fd3f4b96b6bde3d2bdd11d12, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682326700761 2023-04-24 08:58:25,603 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 2b67e509c97f425bbc0f21a0c2f30542, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682326703115 2023-04-24 08:58:25,604 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4c1d6acf22b04e43a87fe396cdbb69c7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682326703115 2023-04-24 08:58:25,604 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting d09093964f7041a78bbd0c7bba277461, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682326703439 2023-04-24 08:58:25,604 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 6fa81489bda647a888fda5ea72aefa03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682326703439 2023-04-24 08:58:25,606 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 948dd94e78524c8da4d49f0b5841e01e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682326704107 2023-04-24 08:58:25,606 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 38f33704f6484b6381ae17dfcbbb7529, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682326704107 2023-04-24 08:58:25,619 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#A#compaction#24 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:25,623 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#B#compaction#25 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:25,651 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:25,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:25,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:25,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:25,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:25,656 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:25,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:25,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:25,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:25,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,666 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/5b8b695c4c544c67b42a1dd8323ae676 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5b8b695c4c544c67b42a1dd8323ae676 2023-04-24 08:58:25,666 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:25,666 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:25,668 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,668 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:25,668 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/c50caec43fdc47d386f61e9b2c23020d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c50caec43fdc47d386f61e9b2c23020d 2023-04-24 08:58:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:25,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:25,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:25,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:25,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:25,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:25,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:25,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:25,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:25,675 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,675 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:25,675 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-24 08:58:25,675 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:25,675 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:25,676 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:25,676 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:25,676 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:25,676 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:25,681 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/A of a1f74229b819199749846ce3feed93fe into 5b8b695c4c544c67b42a1dd8323ae676(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:25,682 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:25,682 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/A, priority=12, startTime=1682326705595; duration=0sec 2023-04-24 08:58:25,682 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:25,682 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:25,683 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/B of a1f74229b819199749846ce3feed93fe into c50caec43fdc47d386f61e9b2c23020d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:25,683 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:25,684 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/B, priority=12, startTime=1682326705595; duration=0sec 2023-04-24 08:58:25,684 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:25,688 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:25,689 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/C is initiating minor compaction (all files) 2023-04-24 08:58:25,689 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/C in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:25,689 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/2a8f5501344243a39c2c45fea7aea6ff, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1f5ff89df86741da8fa4e3435be71359, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/009863d17dfc4018a95a361606e63cce, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3c90d70e96bd4a9ab83ade5f8c35cf8f] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=46.9 K 2023-04-24 08:58:25,690 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 2a8f5501344243a39c2c45fea7aea6ff, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682326700761 2023-04-24 08:58:25,690 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1f5ff89df86741da8fa4e3435be71359, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1682326703115 2023-04-24 08:58:25,691 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 009863d17dfc4018a95a361606e63cce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1682326703439 2023-04-24 08:58:25,691 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 3c90d70e96bd4a9ab83ade5f8c35cf8f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682326704107 2023-04-24 08:58:25,712 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#C#compaction#27 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:25,727 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/f7f1a2851eae4ccb9c27ba4566060fe7 2023-04-24 08:58:25,738 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/244172ef37b24a23babc3be750bdc3c0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/244172ef37b24a23babc3be750bdc3c0 2023-04-24 08:58:25,749 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/C of a1f74229b819199749846ce3feed93fe into 244172ef37b24a23babc3be750bdc3c0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:25,749 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:25,750 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/C, priority=12, startTime=1682326705595; duration=0sec 2023-04-24 08:58:25,750 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:25,769 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/c98530b9681848f38ea06d31d94fa0f7 2023-04-24 08:58:25,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:25,895 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:25,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326765911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326765912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326765941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326765941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:25,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:25,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326765960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326766016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326766016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,205 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/1a9558441a9b4be48ede79ec0c9ada96 2023-04-24 08:58:26,211 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/f7f1a2851eae4ccb9c27ba4566060fe7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f7f1a2851eae4ccb9c27ba4566060fe7 2023-04-24 08:58:26,219 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f7f1a2851eae4ccb9c27ba4566060fe7, entries=150, sequenceid=229, filesize=11.8 K 2023-04-24 08:58:26,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/c98530b9681848f38ea06d31d94fa0f7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c98530b9681848f38ea06d31d94fa0f7 2023-04-24 08:58:26,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326766219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326766220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,238 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c98530b9681848f38ea06d31d94fa0f7, entries=150, sequenceid=229, filesize=11.8 K 2023-04-24 08:58:26,240 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/1a9558441a9b4be48ede79ec0c9ada96 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1a9558441a9b4be48ede79ec0c9ada96 2023-04-24 08:58:26,249 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1a9558441a9b4be48ede79ec0c9ada96, entries=150, sequenceid=229, filesize=11.8 K 2023-04-24 08:58:26,251 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for a1f74229b819199749846ce3feed93fe in 576ms, sequenceid=229, compaction requested=false 2023-04-24 08:58:26,251 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:26,251 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:26,251 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:26,252 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:26,252 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,252 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:26,252 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:26,283 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:26,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:26,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:26,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:26,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45691520[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:26,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:26,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:26,291 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:26,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:26,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:26,291 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,292 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:26,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:26,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:26,299 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:26,300 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:26,300 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:26,299 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:26,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:26,306 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:26,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:26,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:26,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:26,307 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:26,316 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,316 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:26,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,316 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:26,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:26,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,318 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:26,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:26,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:26,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:26,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:26,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:26,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:26,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:26,328 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:26,328 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,328 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:26,328 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:26,328 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:26,328 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:26,328 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,428 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:26,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:26,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:26,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:26,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:26,432 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:26,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:26,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:26,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:26,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:26,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,441 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:26,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:26,442 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,442 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:26,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:26,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:26,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:26,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:26,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:26,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:26,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:26,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:26,447 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:26,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:26,447 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:26,447 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:58:26,448 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:26,448 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:26,448 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:26,448 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:26,448 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:26,449 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:26,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:26,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:26,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326706565, max timestamp=1682326706582Num uniques 859; 2023-04-24 08:58:26,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:26,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326706565, max timestamp=1682326706582Num uniques 859; 2023-04-24 08:58:26,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326706565, max timestamp=1682326706582Num uniques 859; 2023-04-24 08:58:26,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:26,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:26,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326766629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326766630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326766736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326766736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,888 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/f6248cd3ea1d40bf841a463b46ac18b9 2023-04-24 08:58:26,909 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/f8eb7f1e8cd14e808e91fb98b66761de 2023-04-24 08:58:26,925 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/5126036c7c6d4b60b330e751c82ee352 2023-04-24 08:58:26,931 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/f6248cd3ea1d40bf841a463b46ac18b9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f6248cd3ea1d40bf841a463b46ac18b9 2023-04-24 08:58:26,938 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f6248cd3ea1d40bf841a463b46ac18b9, entries=150, sequenceid=242, filesize=11.8 K 2023-04-24 08:58:26,941 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/f8eb7f1e8cd14e808e91fb98b66761de as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f8eb7f1e8cd14e808e91fb98b66761de 2023-04-24 08:58:26,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326766941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326766941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,950 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f8eb7f1e8cd14e808e91fb98b66761de, entries=150, sequenceid=242, filesize=11.8 K 2023-04-24 08:58:26,953 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/5126036c7c6d4b60b330e751c82ee352 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/5126036c7c6d4b60b330e751c82ee352 2023-04-24 08:58:26,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326766953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:26,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326766953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,965 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/5126036c7c6d4b60b330e751c82ee352, entries=150, sequenceid=242, filesize=11.8 K 2023-04-24 08:58:26,967 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a1f74229b819199749846ce3feed93fe in 520ms, sequenceid=242, compaction requested=true 2023-04-24 08:58:26,967 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:26,968 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:26,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:26,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:26,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:26,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:26,968 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:26,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:26,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-24 08:58:26,971 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:26,971 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 08:58:26,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:26,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 08:58:26,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:26,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 08:58:26,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/bc85857fbf544118a2deade60507f302 2023-04-24 08:58:27,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:27,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326767011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,016 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,017 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,017 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:27,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@747cc867[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:27,018 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:27,018 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,024 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,025 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,025 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:27,025 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:27,025 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:27,025 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:27,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,033 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,033 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,033 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:27,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:27,035 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:27,035 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:27,035 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:27,035 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:27,035 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:27,041 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,041 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,041 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,042 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,043 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,070 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:27,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:27,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,071 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:27,071 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:27,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:27,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326767116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,171 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:27,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:27,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:27,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:27,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:27,175 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:27,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:27,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:27,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,183 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:27,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:27,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:27,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:27,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:27,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:27,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:27,185 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,185 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:27,185 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:27,185 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:27,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:27,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:27,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:27,191 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,191 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,192 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,192 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@544598e9[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:27,193 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:27,193 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,200 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:27,200 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:27,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:27,200 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:27,208 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:27,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:27,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:27,210 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:27,210 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:27,210 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:27,210 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:27,210 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:27,216 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,216 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,216 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,218 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,224 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:27,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:27,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,225 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:27,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:27,225 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,225 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:27,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:27,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326767247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:27,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326767248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:27,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326767319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,325 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:27,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:27,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:27,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:27,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:27,331 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:27,331 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,331 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:27,331 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:27,332 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,332 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,341 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:27,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:27,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:27,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:27,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:27,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:27,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:27,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:27,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:27,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:27,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:27,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:27,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:27,350 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,350 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,350 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,350 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:27,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a75d755[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:27,353 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:27,353 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,358 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,358 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:27,358 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:27,358 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:27,358 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,358 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:27,366 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,366 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:27,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,366 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,367 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:27,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:27,368 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:27,368 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:27,368 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:27,368 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:27,368 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:27,374 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,375 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,375 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,377 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,383 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:27,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:27,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:27,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,384 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,384 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:27,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:27,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/9ee8e6b4c2d14a63849ab15ed862c24a 2023-04-24 08:58:27,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:27,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:27,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:27,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:27,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:27,488 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:27,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:27,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:27,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,491 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:27,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:27,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:27,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:27,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:27,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:27,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:27,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:27,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:27,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:27,497 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:27,497 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,497 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:27,497 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:27,507 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,507 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,509 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,509 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:27,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1738012e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:27,510 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:27,510 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,516 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,517 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:27,517 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:27,517 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:27,517 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,517 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:27,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:27,525 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:27,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:27,528 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:27,528 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:27,528 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:27,528 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:27,528 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:27,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,536 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,557 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:27,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:27,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,557 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:27,557 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:27,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57328 deadline: 1682326767624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,657 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:27,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:27,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:27,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:27,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:27,661 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:27,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:27,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:27,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,675 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,675 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,675 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,675 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:27,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:27,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:27,677 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:27,677 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,677 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:27,677 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:27,677 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:27,677 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,677 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:27,677 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:27,678 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:27,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:27,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:27,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:27,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,683 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,683 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,683 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:27,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@515dc5e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:27,685 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:27,685 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,692 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:27,692 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:27,692 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:27,693 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,693 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:27,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:27,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:27,700 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:27,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:27,702 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:27,702 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:27,702 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:27,702 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:27,702 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:27,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,709 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,712 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,716 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,717 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,717 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,717 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,717 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,717 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:27,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:27,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,717 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:27,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:27,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326767749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:27,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326767755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:27,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:27,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:27,821 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:27,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:27,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:27,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,824 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,825 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,825 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/76fef896c21d47e490ea50e31d1339bc 2023-04-24 08:58:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:27,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:27,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:27,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:27,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:27,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:27,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:27,827 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,827 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:27,827 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:27,827 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,827 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:27,827 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:27,827 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,827 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:27,827 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:27,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/bc85857fbf544118a2deade60507f302 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/bc85857fbf544118a2deade60507f302 2023-04-24 08:58:27,833 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,834 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,834 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:27,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7def9ec5[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:27,836 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:27,836 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,839 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/bc85857fbf544118a2deade60507f302, entries=200, sequenceid=280, filesize=14.3 K 2023-04-24 08:58:27,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,842 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:27,842 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:27,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/9ee8e6b4c2d14a63849ab15ed862c24a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/9ee8e6b4c2d14a63849ab15ed862c24a 2023-04-24 08:58:27,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,843 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:27,842 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:27,850 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/9ee8e6b4c2d14a63849ab15ed862c24a, entries=150, sequenceid=280, filesize=11.9 K 2023-04-24 08:58:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/76fef896c21d47e490ea50e31d1339bc as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/76fef896c21d47e490ea50e31d1339bc 2023-04-24 08:58:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:27,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:27,854 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:27,854 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:27,854 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:27,854 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:27,854 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:27,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:27,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,859 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,861 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/76fef896c21d47e490ea50e31d1339bc, entries=150, sequenceid=280, filesize=11.9 K 2023-04-24 08:58:27,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for a1f74229b819199749846ce3feed93fe in 894ms, sequenceid=280, compaction requested=true 2023-04-24 08:58:27,865 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:27,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:27,865 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:27,865 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:27,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:27,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:27,866 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,866 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,866 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:27,866 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:27,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:27,866 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,866 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:27,867 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,867 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:27,867 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,867 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:27,867 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:27,867 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:27,867 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48568 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:27,867 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51008 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:27,867 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/B is initiating minor compaction (all files) 2023-04-24 08:58:27,867 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/A is initiating minor compaction (all files) 2023-04-24 08:58:27,867 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/B in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,867 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/A in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,868 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c50caec43fdc47d386f61e9b2c23020d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c98530b9681848f38ea06d31d94fa0f7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f8eb7f1e8cd14e808e91fb98b66761de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/9ee8e6b4c2d14a63849ab15ed862c24a] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=47.4 K 2023-04-24 08:58:27,868 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5b8b695c4c544c67b42a1dd8323ae676, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f7f1a2851eae4ccb9c27ba4566060fe7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f6248cd3ea1d40bf841a463b46ac18b9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/bc85857fbf544118a2deade60507f302] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=49.8 K 2023-04-24 08:58:27,868 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c50caec43fdc47d386f61e9b2c23020d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682326704107 2023-04-24 08:58:27,870 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 5b8b695c4c544c67b42a1dd8323ae676, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682326704107 2023-04-24 08:58:27,871 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c98530b9681848f38ea06d31d94fa0f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682326704806 2023-04-24 08:58:27,871 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f7f1a2851eae4ccb9c27ba4566060fe7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682326704806 2023-04-24 08:58:27,871 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f8eb7f1e8cd14e808e91fb98b66761de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1682326705902 2023-04-24 08:58:27,871 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f6248cd3ea1d40bf841a463b46ac18b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1682326705902 2023-04-24 08:58:27,872 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 9ee8e6b4c2d14a63849ab15ed862c24a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682326706605 2023-04-24 08:58:27,872 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting bc85857fbf544118a2deade60507f302, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682326706605 2023-04-24 08:58:27,905 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#A#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:27,915 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#B#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:27,943 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/9f3e70d4d0ec462fbc1b597c3459e533 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9f3e70d4d0ec462fbc1b597c3459e533 2023-04-24 08:58:27,950 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/A of a1f74229b819199749846ce3feed93fe into 9f3e70d4d0ec462fbc1b597c3459e533(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:27,950 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:27,950 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/A, priority=12, startTime=1682326707865; duration=0sec 2023-04-24 08:58:27,950 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:27,950 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:27,952 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48568 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:27,952 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/C is initiating minor compaction (all files) 2023-04-24 08:58:27,952 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/C in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,952 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/244172ef37b24a23babc3be750bdc3c0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1a9558441a9b4be48ede79ec0c9ada96, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/5126036c7c6d4b60b330e751c82ee352, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/76fef896c21d47e490ea50e31d1339bc] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=47.4 K 2023-04-24 08:58:27,953 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 244172ef37b24a23babc3be750bdc3c0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1682326704107 2023-04-24 08:58:27,953 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1a9558441a9b4be48ede79ec0c9ada96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1682326704806 2023-04-24 08:58:27,953 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 5126036c7c6d4b60b330e751c82ee352, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1682326705902 2023-04-24 08:58:27,954 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 76fef896c21d47e490ea50e31d1339bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682326706605 2023-04-24 08:58:27,967 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:27,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:27,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:27,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:27,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:27,972 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:27,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:27,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:27,973 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,973 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,975 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#C#compaction#38 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:27,983 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:27,983 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:27,983 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:27,983 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:27,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:27,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:27,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:27,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:27,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:27,988 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:27,988 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:27,988 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:27,988 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:27,988 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 08:58:27,988 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:27,989 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:27,989 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:27,989 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:27,989 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:27,989 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:28,024 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/16148ab5615046e0a72def21c93f49ae as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/16148ab5615046e0a72def21c93f49ae 2023-04-24 08:58:28,033 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/C of a1f74229b819199749846ce3feed93fe into 16148ab5615046e0a72def21c93f49ae(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:28,033 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:28,033 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/C, priority=12, startTime=1682326707865; duration=0sec 2023-04-24 08:58:28,033 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:28,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:28,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326708167, max timestamp=1682326708179Num uniques 859; 2023-04-24 08:58:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326708167, max timestamp=1682326708179Num uniques 859; 2023-04-24 08:58:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326708167, max timestamp=1682326708179Num uniques 859; 2023-04-24 08:58:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:28,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:28,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326768229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:28,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:28,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326768332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:28,340 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/6a530dca3e9540559ea821db5c857a9c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6a530dca3e9540559ea821db5c857a9c 2023-04-24 08:58:28,348 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a1f74229b819199749846ce3feed93fe/B of a1f74229b819199749846ce3feed93fe into 6a530dca3e9540559ea821db5c857a9c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:28,348 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:28,348 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/B, priority=12, startTime=1682326707865; duration=0sec 2023-04-24 08:58:28,348 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:28,443 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/532a2bab2fe843e0874807e328102fb0 2023-04-24 08:58:28,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:28,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326768536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:28,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:28,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326768755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:28,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326768765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:28,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:28,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326768842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:28,860 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/f275bc14c6a94b9d9721072339ed3b75 2023-04-24 08:58:28,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:28,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326768962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:28,967 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:28,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:28,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326768977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:28,980 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:29,278 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/3e5a0b16f077452a8c831d18a19e0776 2023-04-24 08:58:29,284 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/532a2bab2fe843e0874807e328102fb0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/532a2bab2fe843e0874807e328102fb0 2023-04-24 08:58:29,289 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/532a2bab2fe843e0874807e328102fb0, entries=100, sequenceid=290, filesize=9.5 K 2023-04-24 08:58:29,291 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/f275bc14c6a94b9d9721072339ed3b75 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f275bc14c6a94b9d9721072339ed3b75 2023-04-24 08:58:29,297 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f275bc14c6a94b9d9721072339ed3b75, entries=100, sequenceid=290, filesize=9.5 K 2023-04-24 08:58:29,298 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/3e5a0b16f077452a8c831d18a19e0776 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3e5a0b16f077452a8c831d18a19e0776 2023-04-24 08:58:29,304 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3e5a0b16f077452a8c831d18a19e0776, entries=100, sequenceid=290, filesize=9.5 K 2023-04-24 08:58:29,305 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for a1f74229b819199749846ce3feed93fe in 1316ms, sequenceid=290, compaction requested=false 2023-04-24 08:58:29,305 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:29,305 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:29,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:29,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:29,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:29,316 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,316 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,316 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,316 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:29,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49943152[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:29,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:29,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,324 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:29,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:29,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:29,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:29,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:29,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:29,335 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:29,335 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:29,335 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:29,335 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:29,336 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:29,341 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,342 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,343 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:29,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:29,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-24 08:58:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 08:58:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:29,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682326708195, max timestamp=1682326709344Num uniques 1068; 2023-04-24 08:58:29,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:29,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-24 08:58:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:29,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 08:58:29,358 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:29,358 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:29,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:29,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:29,359 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,359 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,359 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,359 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,359 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,359 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,359 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:29,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:29,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:29,363 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:29,363 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:29,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/4284046c64db4bec9cd9cc2aac563d65 2023-04-24 08:58:29,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:29,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326769398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,463 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:29,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:29,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:29,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:29,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:29,468 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:29,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:29,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:29,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:29,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,491 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:29,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:29,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:29,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:29,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:29,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:29,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:29,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:29,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:29,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:29,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:29,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:29,497 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:29,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326769504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,508 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,508 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,508 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,508 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:29,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef2783f[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:29,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:29,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,516 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:29,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:29,516 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,517 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:29,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:29,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:29,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:29,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:29,525 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:29,529 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:29,529 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:29,529 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:29,529 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:29,529 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:29,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:29,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,535 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,536 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:29,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:29,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:29,542 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:29,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:29,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:29,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:29,542 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,542 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,542 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:29,542 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:29,542 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,542 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:29,643 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:29,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:29,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:29,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:29,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:29,646 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:29,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:29,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:29,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:29,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,649 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,649 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:29,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:29,650 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,650 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:29,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:29,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:29,651 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:29,651 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:29,651 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:29,651 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:29,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:29,658 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:29,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@352ea093[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:29,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:29,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,666 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,666 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,666 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:29,666 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:29,666 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:29,666 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:29,674 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:29,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:29,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:29,677 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:29,677 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:29,677 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:29,677 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:29,677 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:29,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,683 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,685 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:29,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:29,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:29,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:29,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,692 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,692 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,692 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,692 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:29,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:29,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:29,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:29,692 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:29,692 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:29,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:29,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326769707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,792 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:29,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:29,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:29,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:29,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:29,795 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:29,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:29,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:29,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:29,796 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,796 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,799 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:29,799 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,800 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,800 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:29,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:29,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:29,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:29,801 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:29,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:29,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:29,804 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:29,804 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:29,804 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,804 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:29,804 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:29,804 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,804 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:29,804 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:29,804 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,804 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:29,804 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:29,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:29,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34b60945[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:29,810 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:29,810 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,816 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,816 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,816 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:29,816 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:29,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:29,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:29,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,824 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,825 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,825 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:29,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:29,827 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:29,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:29,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:29,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:29,827 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:29,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/415e283334d84f1fa41f3438dc6bfdf1 2023-04-24 08:58:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,835 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:29,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:29,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:29,841 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:29,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:29,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:29,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:29,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,842 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:29,842 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:29,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:29,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:29,942 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:29,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:29,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:29,945 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:29,945 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:29,945 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:29,945 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:29,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,949 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:29,949 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:29,950 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,950 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:29,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:29,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:29,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:29,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:29,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:29,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:29,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:29,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:29,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:29,951 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,951 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:29,951 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:29,951 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:29,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:29,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:29,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:29,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:29,975 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,975 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:29,976 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,976 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:29,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:29,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c40a020[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:29,981 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:29,981 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,991 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:29,991 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:29,991 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:29,991 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:29,991 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:29,991 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:30,000 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:30,000 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,000 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,000 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:30,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:30,002 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:30,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:30,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:30,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:30,003 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:30,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:30,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:30,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326770010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,025 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:30,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,025 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:30,025 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:30,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:30,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:30,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:30,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:30,125 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:30,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:30,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:30,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:30,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:30,129 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:30,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:30,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:30,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:30,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,133 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:30,133 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:30,133 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,133 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:30,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:30,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:30,134 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:30,134 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:30,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:30,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:30,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:30,135 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,135 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:30,135 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:30,135 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:30,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:30,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:30,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:30,141 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,141 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:30,143 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:30,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51fe1817[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:30,143 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,149 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,150 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:30,150 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:30,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:30,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:30,158 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,158 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,158 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,158 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:30,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:30,160 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:30,160 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:30,160 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:30,160 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:30,160 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:30,166 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,168 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,174 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,174 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:30,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:30,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:30,175 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:30,175 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:30,176 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,176 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:30,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,176 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:30,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:30,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:30,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:30,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/1851648009ba478691a9c9bb6f845382 2023-04-24 08:58:30,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/4284046c64db4bec9cd9cc2aac563d65 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4284046c64db4bec9cd9cc2aac563d65 2023-04-24 08:58:30,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4284046c64db4bec9cd9cc2aac563d65, entries=200, sequenceid=330, filesize=14.3 K 2023-04-24 08:58:30,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/415e283334d84f1fa41f3438dc6bfdf1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/415e283334d84f1fa41f3438dc6bfdf1 2023-04-24 08:58:30,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/415e283334d84f1fa41f3438dc6bfdf1, entries=150, sequenceid=330, filesize=11.9 K 2023-04-24 08:58:30,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/1851648009ba478691a9c9bb6f845382 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1851648009ba478691a9c9bb6f845382 2023-04-24 08:58:30,275 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:30,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:30,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1851648009ba478691a9c9bb6f845382, entries=150, sequenceid=330, filesize=11.9 K 2023-04-24 08:58:30,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:30,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~302.31 KB/309568, currentSize=73.80 KB/75570 for a1f74229b819199749846ce3feed93fe in 924ms, sequenceid=330, compaction requested=true 2023-04-24 08:58:30,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:30,278 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:30,278 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:30,278 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:30,278 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:30,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:30,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:30,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:30,281 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:30,281 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:30,281 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:30,281 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:30,281 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:30,281 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/A is initiating minor compaction (all files) 2023-04-24 08:58:30,281 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/B is initiating minor compaction (all files) 2023-04-24 08:58:30,281 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/A in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,281 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/B in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,281 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6a530dca3e9540559ea821db5c857a9c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f275bc14c6a94b9d9721072339ed3b75, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/415e283334d84f1fa41f3438dc6bfdf1] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=33.7 K 2023-04-24 08:58:30,281 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:30,281 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9f3e70d4d0ec462fbc1b597c3459e533, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/532a2bab2fe843e0874807e328102fb0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4284046c64db4bec9cd9cc2aac563d65] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=36.1 K 2023-04-24 08:58:30,282 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,282 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,282 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 6a530dca3e9540559ea821db5c857a9c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682326706605 2023-04-24 08:58:30,282 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 9f3e70d4d0ec462fbc1b597c3459e533, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682326706605 2023-04-24 08:58:30,282 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f275bc14c6a94b9d9721072339ed3b75, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682326707005 2023-04-24 08:58:30,283 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 532a2bab2fe843e0874807e328102fb0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682326707005 2023-04-24 08:58:30,283 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 415e283334d84f1fa41f3438dc6bfdf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682326708195 2023-04-24 08:58:30,283 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 4284046c64db4bec9cd9cc2aac563d65, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682326708195 2023-04-24 08:58:30,290 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#B#compaction#45 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:30,291 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:30,291 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:30,293 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,293 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:30,294 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#A#compaction#46 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:30,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:30,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:30,295 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:30,296 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:30,296 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:30,296 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:30,296 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:30,296 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,296 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:30,296 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 08:58:30,297 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:30,297 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:30,297 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:30,297 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:30,297 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:30,297 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:30,322 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/b6249aca37534396a1213a9a5e4dd699 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b6249aca37534396a1213a9a5e4dd699 2023-04-24 08:58:30,323 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/c949cba834bd41159a12fa877dfeca70 2023-04-24 08:58:30,329 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/A of a1f74229b819199749846ce3feed93fe into b6249aca37534396a1213a9a5e4dd699(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:30,329 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:30,329 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/A, priority=13, startTime=1682326710278; duration=0sec 2023-04-24 08:58:30,329 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:30,329 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:30,330 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:30,330 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/C is initiating minor compaction (all files) 2023-04-24 08:58:30,330 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/C in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,330 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/16148ab5615046e0a72def21c93f49ae, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3e5a0b16f077452a8c831d18a19e0776, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1851648009ba478691a9c9bb6f845382] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=33.7 K 2023-04-24 08:58:30,331 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 16148ab5615046e0a72def21c93f49ae, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682326706605 2023-04-24 08:58:30,331 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 3e5a0b16f077452a8c831d18a19e0776, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682326707005 2023-04-24 08:58:30,331 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1851648009ba478691a9c9bb6f845382, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682326708195 2023-04-24 08:58:30,338 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#C#compaction#49 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:30,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:30,515 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:30,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:30,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326770560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:30,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326770663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,728 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/8d85db58e49c41d7af6aa4f57687124d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8d85db58e49c41d7af6aa4f57687124d 2023-04-24 08:58:30,734 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/B of a1f74229b819199749846ce3feed93fe into 8d85db58e49c41d7af6aa4f57687124d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:30,734 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:30,734 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/B, priority=13, startTime=1682326710278; duration=0sec 2023-04-24 08:58:30,734 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:30,749 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/14038f9e20fb4481a33636e00c9d2a03 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/14038f9e20fb4481a33636e00c9d2a03 2023-04-24 08:58:30,750 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/64ce7026219f48efb05afaf74ea70579 2023-04-24 08:58:30,755 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/C of a1f74229b819199749846ce3feed93fe into 14038f9e20fb4481a33636e00c9d2a03(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:30,755 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:30,755 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/C, priority=13, startTime=1682326710279; duration=0sec 2023-04-24 08:58:30,755 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:30,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:30,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326770767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,769 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:30,769 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/c42e22ac426841079430be2380e5e5d1 2023-04-24 08:58:30,775 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/c949cba834bd41159a12fa877dfeca70 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/c949cba834bd41159a12fa877dfeca70 2023-04-24 08:58:30,781 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/c949cba834bd41159a12fa877dfeca70, entries=150, sequenceid=344, filesize=11.9 K 2023-04-24 08:58:30,782 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/64ce7026219f48efb05afaf74ea70579 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/64ce7026219f48efb05afaf74ea70579 2023-04-24 08:58:30,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:30,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326770782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,784 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:30,789 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/64ce7026219f48efb05afaf74ea70579, entries=150, sequenceid=344, filesize=11.9 K 2023-04-24 08:58:30,790 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/c42e22ac426841079430be2380e5e5d1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c42e22ac426841079430be2380e5e5d1 2023-04-24 08:58:30,795 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c42e22ac426841079430be2380e5e5d1, entries=150, sequenceid=344, filesize=11.9 K 2023-04-24 08:58:30,795 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a1f74229b819199749846ce3feed93fe in 499ms, sequenceid=344, compaction requested=false 2023-04-24 08:58:30,796 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:30,796 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:30,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:30,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:30,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:30,799 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,799 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,800 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,800 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:30,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11941083[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:30,801 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:30,801 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,809 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:30,809 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:30,809 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:30,809 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,809 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:30,816 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,816 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,816 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:30,816 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:30,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:30,819 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:30,819 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:30,819 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:30,819 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:30,819 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:30,825 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,825 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,825 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,836 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:30,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:30,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,841 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:30,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:30,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:30,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:30,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,842 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:30,842 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,842 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:30,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:30,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:30,867 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 08:58:30,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:30,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:30,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:30,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:30,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:30,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:30,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/76535ca8c0c347db9d53a0470613fdd9 2023-04-24 08:58:30,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:30,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326770904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,942 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:30,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:30,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:30,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:30,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:30,945 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:30,945 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:30,945 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:30,945 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:30,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,949 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:30,950 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:30,950 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,950 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:30,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:30,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:30,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:30,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:30,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:30,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:30,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:30,951 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:30,951 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,951 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:30,952 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:30,952 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:30,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:30,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:30,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:30,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:30,958 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:30,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@156cdfd5[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:30,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:30,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,966 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:30,966 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,966 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:30,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:30,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:30,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:30,992 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,992 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:30,993 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:30,993 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:30,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:30,995 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:30,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:30,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:30,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:30,995 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:31,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,008 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:31,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,010 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:31,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326771008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,016 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:31,016 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:31,017 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:31,017 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:31,017 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:31,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:31,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:31,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,117 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:31,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:31,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:31,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:31,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:31,120 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:31,120 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:31,120 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:31,120 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:31,120 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,120 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,124 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:31,124 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:31,125 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,125 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:31,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:31,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:31,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:31,126 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,126 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:31,126 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:31,126 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:31,126 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:31,133 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,133 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,133 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,133 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:31,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c742cf8[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:31,134 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:31,135 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,141 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,142 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,142 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:31,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:31,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:31,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:31,149 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:31,150 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:31,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:31,152 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:31,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:31,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:31,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:31,152 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:31,158 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,158 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:31,158 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,168 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,169 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:31,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:31,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:31,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:31,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:31,175 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,175 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,175 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:31,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:31,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326771216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,275 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:31,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:31,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:31,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:31,278 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:31,279 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:31,279 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:31,279 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:31,279 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,279 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,283 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:31,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:31,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:31,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:31,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:31,284 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:31,284 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:31,284 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:31,284 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:31,284 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:31,284 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,285 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:31,285 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:31,285 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,285 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:31,285 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:31,285 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,285 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:31,285 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:31,291 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:31,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6babc1[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:31,293 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:31,293 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,299 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,300 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:31,300 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:31,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:31,300 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:31,308 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:31,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:31,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:31,310 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:31,310 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:31,310 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:31,310 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:31,310 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:31,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/a9c284ea6436432392cc0c693d4cc1a7 2023-04-24 08:58:31,316 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,316 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:31,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,316 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:31,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:31,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:31,326 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:31,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:31,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:31,326 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:31,326 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:31,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:31,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:31,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:31,380 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/9fe22a630c114c0b9486dd83c2f1c135 2023-04-24 08:58:31,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/76535ca8c0c347db9d53a0470613fdd9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/76535ca8c0c347db9d53a0470613fdd9 2023-04-24 08:58:31,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/76535ca8c0c347db9d53a0470613fdd9, entries=200, sequenceid=366, filesize=14.3 K 2023-04-24 08:58:31,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/a9c284ea6436432392cc0c693d4cc1a7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/a9c284ea6436432392cc0c693d4cc1a7 2023-04-24 08:58:31,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/a9c284ea6436432392cc0c693d4cc1a7, entries=150, sequenceid=366, filesize=11.9 K 2023-04-24 08:58:31,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/9fe22a630c114c0b9486dd83c2f1c135 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/9fe22a630c114c0b9486dd83c2f1c135 2023-04-24 08:58:31,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/9fe22a630c114c0b9486dd83c2f1c135, entries=150, sequenceid=366, filesize=11.9 K 2023-04-24 08:58:31,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a1f74229b819199749846ce3feed93fe in 557ms, sequenceid=366, compaction requested=true 2023-04-24 08:58:31,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:31,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:31,424 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:31,424 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:31,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:31,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:31,425 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:31,425 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:31,426 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/B is initiating minor compaction (all files) 2023-04-24 08:58:31,426 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/A is initiating minor compaction (all files) 2023-04-24 08:58:31,426 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/B in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,426 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/A in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,426 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8d85db58e49c41d7af6aa4f57687124d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/64ce7026219f48efb05afaf74ea70579, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/a9c284ea6436432392cc0c693d4cc1a7] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=36.2 K 2023-04-24 08:58:31,426 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b6249aca37534396a1213a9a5e4dd699, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/c949cba834bd41159a12fa877dfeca70, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/76535ca8c0c347db9d53a0470613fdd9] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=38.6 K 2023-04-24 08:58:31,426 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:31,426 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 8d85db58e49c41d7af6aa4f57687124d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682326708195 2023-04-24 08:58:31,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:31,427 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting b6249aca37534396a1213a9a5e4dd699, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682326708195 2023-04-24 08:58:31,427 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 64ce7026219f48efb05afaf74ea70579, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682326709383 2023-04-24 08:58:31,427 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c949cba834bd41159a12fa877dfeca70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682326709383 2023-04-24 08:58:31,427 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a9c284ea6436432392cc0c693d4cc1a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682326710555 2023-04-24 08:58:31,428 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 76535ca8c0c347db9d53a0470613fdd9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682326710546 2023-04-24 08:58:31,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:31,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:31,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:31,432 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:31,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:31,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:31,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:31,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,432 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,436 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#A#compaction#54 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:31,437 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#B#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:31,441 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:31,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:31,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:31,442 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,442 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:31,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:31,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:31,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:31,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:31,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:31,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:31,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:31,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:31,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:31,445 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 08:58:31,445 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:31,445 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:31,445 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:31,445 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:31,445 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:31,445 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:31,450 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/0e8e4a8ca8e74205b645b1db9314b0c5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0e8e4a8ca8e74205b645b1db9314b0c5 2023-04-24 08:58:31,459 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/A of a1f74229b819199749846ce3feed93fe into 0e8e4a8ca8e74205b645b1db9314b0c5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:31,460 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:31,460 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/A, priority=13, startTime=1682326711424; duration=0sec 2023-04-24 08:58:31,460 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:31,460 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:31,464 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:31,465 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/C is initiating minor compaction (all files) 2023-04-24 08:58:31,465 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/C in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,465 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/14038f9e20fb4481a33636e00c9d2a03, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c42e22ac426841079430be2380e5e5d1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/9fe22a630c114c0b9486dd83c2f1c135] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=36.2 K 2023-04-24 08:58:31,466 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 14038f9e20fb4481a33636e00c9d2a03, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1682326708195 2023-04-24 08:58:31,466 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c42e22ac426841079430be2380e5e5d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682326709383 2023-04-24 08:58:31,466 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 9fe22a630c114c0b9486dd83c2f1c135, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682326710555 2023-04-24 08:58:31,472 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/e6b2e511c20346e29ec94247d909b3bb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/e6b2e511c20346e29ec94247d909b3bb 2023-04-24 08:58:31,475 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/ab6a456ba68e439c9bf51fa07aa46de2 2023-04-24 08:58:31,483 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/B of a1f74229b819199749846ce3feed93fe into e6b2e511c20346e29ec94247d909b3bb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:31,483 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:31,483 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/B, priority=13, startTime=1682326711424; duration=0sec 2023-04-24 08:58:31,483 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:31,483 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#C#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:31,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:31,522 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:31,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:31,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326771568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:31,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326771670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:31,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326771874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,901 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/c4e74b41da754b7dbbb399f967c5b0c3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c4e74b41da754b7dbbb399f967c5b0c3 2023-04-24 08:58:31,906 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/7467f51ebf5b4fdb99004b6fe3cc6208 2023-04-24 08:58:31,908 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/C of a1f74229b819199749846ce3feed93fe into c4e74b41da754b7dbbb399f967c5b0c3(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:31,908 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:31,908 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/C, priority=13, startTime=1682326711424; duration=0sec 2023-04-24 08:58:31,908 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:31,944 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/f8370e68f975414c9bae5876dbfa61f0 2023-04-24 08:58:31,949 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/ab6a456ba68e439c9bf51fa07aa46de2 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/ab6a456ba68e439c9bf51fa07aa46de2 2023-04-24 08:58:31,954 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/ab6a456ba68e439c9bf51fa07aa46de2, entries=150, sequenceid=379, filesize=11.9 K 2023-04-24 08:58:31,955 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/7467f51ebf5b4fdb99004b6fe3cc6208 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7467f51ebf5b4fdb99004b6fe3cc6208 2023-04-24 08:58:31,961 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7467f51ebf5b4fdb99004b6fe3cc6208, entries=150, sequenceid=379, filesize=11.9 K 2023-04-24 08:58:31,962 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/f8370e68f975414c9bae5876dbfa61f0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f8370e68f975414c9bae5876dbfa61f0 2023-04-24 08:58:31,968 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f8370e68f975414c9bae5876dbfa61f0, entries=150, sequenceid=379, filesize=11.9 K 2023-04-24 08:58:31,969 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a1f74229b819199749846ce3feed93fe in 524ms, sequenceid=379, compaction requested=false 2023-04-24 08:58:31,969 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:31,970 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:31,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:31,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:31,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:31,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:31,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,975 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,975 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,975 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:31,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4aaee7[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:31,977 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:31,977 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,983 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:31,983 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:31,983 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:31,983 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:31,984 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,984 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:31,992 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:31,992 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,992 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:31,992 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:31,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:31,994 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:31,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:31,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:31,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:31,995 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:32,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:32,018 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:32,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:32,021 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:32,033 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:32,033 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:32,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:32,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:32,034 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:32,034 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:32,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:32,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:32,134 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:32,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:32,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:32,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:32,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:32,139 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:32,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:32,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:32,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:32,140 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,140 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:32,149 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:32,149 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:32,150 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,150 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:32,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:32,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:32,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:32,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:32,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:32,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:32,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:32,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:32,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:32,151 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:32,152 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:32,152 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 08:58:32,152 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:32,152 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:32,152 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:32,152 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:32,152 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:32,152 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:32,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:32,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:32,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:32,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326772210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:32,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326772315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:32,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326772519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,572 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/028a4d48a453446785418c5d1c07114f 2023-04-24 08:58:32,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:32,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326772823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:32,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57364 deadline: 1682326772995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:32,999 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:33,000 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/74e7dfe1f6454f9d87f0e7788210cdf7 2023-04-24 08:58:33,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:33,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57348 deadline: 1682326772997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,001 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:33,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:33,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326773324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,413 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/e50ab579929a49a7b6f2b5a8cc913618 2023-04-24 08:58:33,418 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/028a4d48a453446785418c5d1c07114f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/028a4d48a453446785418c5d1c07114f 2023-04-24 08:58:33,423 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/028a4d48a453446785418c5d1c07114f, entries=150, sequenceid=401, filesize=11.9 K 2023-04-24 08:58:33,424 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/74e7dfe1f6454f9d87f0e7788210cdf7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/74e7dfe1f6454f9d87f0e7788210cdf7 2023-04-24 08:58:33,428 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/74e7dfe1f6454f9d87f0e7788210cdf7, entries=150, sequenceid=401, filesize=11.9 K 2023-04-24 08:58:33,430 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/e50ab579929a49a7b6f2b5a8cc913618 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/e50ab579929a49a7b6f2b5a8cc913618 2023-04-24 08:58:33,434 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/e50ab579929a49a7b6f2b5a8cc913618, entries=150, sequenceid=401, filesize=11.9 K 2023-04-24 08:58:33,435 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a1f74229b819199749846ce3feed93fe in 1283ms, sequenceid=401, compaction requested=true 2023-04-24 08:58:33,435 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:33,435 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:33,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:33,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:33,436 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,436 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:33,436 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:33,441 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:33,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:33,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:33,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:33,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:33,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f2b50cc[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:33,443 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:33,443 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:33,449 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:33,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:33,450 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:33,450 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:33,450 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:33,450 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,450 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:33,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:33,458 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:33,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:33,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:33,460 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:33,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:33,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:33,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:33,461 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:33,466 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,466 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:33,466 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:33,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:33,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:33,467 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:33,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:33,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:33,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:33,474 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:33,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:33,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:33,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:33,475 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:33,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:33,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:33,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:33,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:33,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:33,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:33,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:33,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:33,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:33,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:33,584 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:33,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:33,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:33,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:33,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:33,591 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:33,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:33,592 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:33,592 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:33,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:33,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:33,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:33,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:33,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:33,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:33,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:33,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:33,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:33,593 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:33,593 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:33,594 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 08:58:33,594 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:33,594 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:33,594 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:33,594 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:33,594 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:33,594 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:33,623 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/0844b7ecbe3442f0974cb82f39d8e62e 2023-04-24 08:58:33,643 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/164817d197cd43a48f032d6ebcbc8cea 2023-04-24 08:58:34,062 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/d5015532e53e45c595de6a1da0ee9feb 2023-04-24 08:58:34,067 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/0844b7ecbe3442f0974cb82f39d8e62e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0844b7ecbe3442f0974cb82f39d8e62e 2023-04-24 08:58:34,073 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0844b7ecbe3442f0974cb82f39d8e62e, entries=150, sequenceid=414, filesize=11.9 K 2023-04-24 08:58:34,074 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/164817d197cd43a48f032d6ebcbc8cea as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/164817d197cd43a48f032d6ebcbc8cea 2023-04-24 08:58:34,078 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/164817d197cd43a48f032d6ebcbc8cea, entries=150, sequenceid=414, filesize=11.9 K 2023-04-24 08:58:34,079 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/d5015532e53e45c595de6a1da0ee9feb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/d5015532e53e45c595de6a1da0ee9feb 2023-04-24 08:58:34,083 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/d5015532e53e45c595de6a1da0ee9feb, entries=150, sequenceid=414, filesize=11.9 K 2023-04-24 08:58:34,084 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for a1f74229b819199749846ce3feed93fe in 490ms, sequenceid=414, compaction requested=true 2023-04-24 08:58:34,084 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:34,084 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,084 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:34,084 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:34,084 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,084 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:34,084 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:34,116 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,116 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,116 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,116 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:34,118 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:34,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4255aa3d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:34,118 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,124 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,125 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:34,125 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:34,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:34,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:34,133 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,133 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:34,133 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,133 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:34,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:34,135 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:34,135 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:34,135 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:34,135 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:34,135 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:34,141 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,141 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,141 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,142 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,143 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,149 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,149 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,149 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,149 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,150 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:34,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:34,150 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,150 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,150 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,150 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,150 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:34,150 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,150 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,250 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:34,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:34,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:34,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:34,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:34,253 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:34,253 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,253 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:34,253 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:34,254 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,254 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,258 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:34,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:34,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:34,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:34,259 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:34,259 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,259 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:34,259 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:34,259 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:34,259 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,259 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:34,259 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:34,259 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,260 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:34,260 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:34,260 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,260 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:34,260 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:34,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,266 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,266 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,266 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:34,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:34,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72d03415[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:34,267 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:34,267 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,274 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,274 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,274 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:34,275 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:34,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:34,275 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:34,283 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:34,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:34,285 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:34,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:34,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:34,286 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:34,286 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:34,291 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,291 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,291 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,292 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,293 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:34,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,320 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:34,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,320 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:34,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,320 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,320 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,320 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,320 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,320 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:34,350 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:58:34,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:34,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:34,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:34,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:34,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:34,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:34,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326714400, max timestamp=1682326714415Num uniques 859; 2023-04-24 08:58:34,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326714400, max timestamp=1682326714415Num uniques 859; 2023-04-24 08:58:34,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:34,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:34,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326714400, max timestamp=1682326714415Num uniques 859; 2023-04-24 08:58:34,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:34,420 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:34,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:34,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:34,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:34,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:34,429 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:34,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:34,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:34,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,433 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,433 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,433 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,433 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:34,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:34,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:34,434 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:34,434 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,434 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:34,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:34,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:34,435 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,435 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:34,435 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:34,435 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:34,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:34,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:34,435 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:34,441 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:34,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b64813e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:34,443 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:34,443 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,449 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,450 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:34,450 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:34,450 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:34,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:34,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:34,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:34,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:34,458 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:34,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:34,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:34,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:34,458 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:34,466 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,466 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,467 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,469 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:34,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:34,474 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:34,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,475 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:34,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:34,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326774479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:34,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:34,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:34,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:34,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:34,578 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:34,578 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,578 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:34,578 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:34,578 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,578 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,583 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,583 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,583 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,583 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:34,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:34,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:34,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326774582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:34,584 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:34,584 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,584 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:34,584 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:34,584 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:34,584 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,585 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:34,585 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:34,585 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,585 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:34,585 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:34,585 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,585 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:34,585 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:34,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,591 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,591 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,591 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:34,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ebc13ae[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:34,593 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:34,593 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,599 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,600 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:34,600 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:34,600 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:34,600 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,600 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:34,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:34,612 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:34,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:34,614 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:34,615 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:34,615 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:34,615 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:34,615 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:34,624 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,624 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,624 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,625 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,626 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:34,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:34,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,633 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:34,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:34,633 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,633 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,634 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,634 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,634 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,634 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,734 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:34,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:34,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:34,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:34,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:34,738 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:34,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:34,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:34,744 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,744 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,750 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,750 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:34,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:34,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:34,751 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:34,751 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,751 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:34,751 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:34,751 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:34,751 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,751 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:34,752 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:34,752 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,752 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:34,752 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:34,752 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,752 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:34,752 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:34,758 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:34,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61575406[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:34,759 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:34,759 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,762 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/536ffb91ffa84c57a46f71bc64f8baa4 2023-04-24 08:58:34,766 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,766 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,766 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:34,767 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:34,767 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:34,767 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:34,774 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,774 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,774 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,775 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:34,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:34,777 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:34,777 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:34,777 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:34,777 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:34,777 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:34,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,783 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,784 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:34,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326774786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:34,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57392 deadline: 1682326774801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,804 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:34,812 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,812 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,812 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,812 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:34,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:34,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:34,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,812 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:34,812 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:34,812 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,812 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,812 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,812 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:34,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57372 deadline: 1682326774817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,822 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:34,912 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:34,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:34,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:34,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:34,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:34,916 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:34,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:34,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:34,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,924 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,924 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:34,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:34,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:34,926 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,926 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:34,926 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:34,926 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:34,926 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:34,933 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,933 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:34,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1254a4a5[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:34,935 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:34,935 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,941 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,941 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:34,941 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:34,941 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:34,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:34,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:34,949 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:34,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:34,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:34,953 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:34,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:34,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:34,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:34,953 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:34,958 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,958 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,958 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:34,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:34,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,960 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,966 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:34,966 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:34,966 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:34,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:34,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:34,966 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,966 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:34,967 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:34,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:34,967 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:34,967 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,967 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:34,967 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:34,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:34,967 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:34,967 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:35,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:35,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:35,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:35,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:35,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:35,071 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:35,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:35,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:35,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:35,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:35,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:35,074 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:35,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:35,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:35,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:35,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:35,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:35,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:35,076 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:35,076 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:35,076 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:35,076 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:35,076 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:35,083 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,083 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:35,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:35,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:35,083 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:35,083 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:35,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:35,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:35,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:35,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:35,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:35,084 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:35,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d66aaa0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:35,084 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:35,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:35,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326775088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,091 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:35,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:35,091 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:35,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:35,091 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:35,091 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:35,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,099 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:35,099 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:35,100 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:35,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:35,102 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:35,102 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:35,102 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:35,102 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:35,102 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:35,108 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,108 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:35,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,108 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:35,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:35,108 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,109 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,116 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,116 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:35,116 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:35,116 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:35,116 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:35,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:35,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:35,116 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:35,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:35,116 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,116 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:35,117 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:35,117 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,117 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:35,117 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:35,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:35,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/50ac784f44754e2997c9b2e01667ebdf 2023-04-24 08:58:35,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/904346432a39489cbe9d261e354a3c08 2023-04-24 08:58:35,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/536ffb91ffa84c57a46f71bc64f8baa4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/536ffb91ffa84c57a46f71bc64f8baa4 2023-04-24 08:58:35,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/536ffb91ffa84c57a46f71bc64f8baa4, entries=200, sequenceid=425, filesize=14.3 K 2023-04-24 08:58:35,217 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:35,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/50ac784f44754e2997c9b2e01667ebdf as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/50ac784f44754e2997c9b2e01667ebdf 2023-04-24 08:58:35,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:35,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:35,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:35,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:35,228 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:35,228 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:35,228 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:35,228 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:35,229 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,229 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:35,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/50ac784f44754e2997c9b2e01667ebdf, entries=150, sequenceid=425, filesize=11.9 K 2023-04-24 08:58:35,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/904346432a39489cbe9d261e354a3c08 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/904346432a39489cbe9d261e354a3c08 2023-04-24 08:58:35,233 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:35,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:35,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:35,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/904346432a39489cbe9d261e354a3c08, entries=150, sequenceid=425, filesize=11.9 K 2023-04-24 08:58:35,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:35,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:35,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a1f74229b819199749846ce3feed93fe in 891ms, sequenceid=425, compaction requested=true 2023-04-24 08:58:35,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:35,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:35,241 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:58:35,241 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:58:35,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:35,241 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:35,243 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:58:35,244 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/B is initiating minor compaction (all files) 2023-04-24 08:58:35,244 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/B in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:35,244 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/e6b2e511c20346e29ec94247d909b3bb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7467f51ebf5b4fdb99004b6fe3cc6208, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/74e7dfe1f6454f9d87f0e7788210cdf7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/164817d197cd43a48f032d6ebcbc8cea, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/50ac784f44754e2997c9b2e01667ebdf] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=60.1 K 2023-04-24 08:58:35,244 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63984 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:58:35,245 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/A is initiating minor compaction (all files) 2023-04-24 08:58:35,245 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/A in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:35,245 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0e8e4a8ca8e74205b645b1db9314b0c5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/ab6a456ba68e439c9bf51fa07aa46de2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/028a4d48a453446785418c5d1c07114f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0844b7ecbe3442f0974cb82f39d8e62e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/536ffb91ffa84c57a46f71bc64f8baa4] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=62.5 K 2023-04-24 08:58:35,245 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e6b2e511c20346e29ec94247d909b3bb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682326710555 2023-04-24 08:58:35,245 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 0e8e4a8ca8e74205b645b1db9314b0c5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682326710555 2023-04-24 08:58:35,245 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7467f51ebf5b4fdb99004b6fe3cc6208, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682326710870 2023-04-24 08:58:35,246 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting ab6a456ba68e439c9bf51fa07aa46de2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682326710870 2023-04-24 08:58:35,246 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 74e7dfe1f6454f9d87f0e7788210cdf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682326711561 2023-04-24 08:58:35,246 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 028a4d48a453446785418c5d1c07114f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682326711561 2023-04-24 08:58:35,246 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 164817d197cd43a48f032d6ebcbc8cea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1682326712189 2023-04-24 08:58:35,247 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 50ac784f44754e2997c9b2e01667ebdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682326714335 2023-04-24 08:58:35,247 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 0844b7ecbe3442f0974cb82f39d8e62e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1682326712189 2023-04-24 08:58:35,247 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 536ffb91ffa84c57a46f71bc64f8baa4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682326714335 2023-04-24 08:58:35,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:35,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:35,249 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:35,249 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:35,249 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:35,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:35,264 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:35,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:35,264 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:35,264 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-24 08:58:35,264 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:35,264 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:35,264 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:35,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:35,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:35,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:35,274 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#A#compaction#69 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:35,277 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#B#compaction#71 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:35,307 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/4793ea1fc01b4a8a9a0a00adc8a3ba21 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4793ea1fc01b4a8a9a0a00adc8a3ba21 2023-04-24 08:58:35,314 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a1f74229b819199749846ce3feed93fe/A of a1f74229b819199749846ce3feed93fe into 4793ea1fc01b4a8a9a0a00adc8a3ba21(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:35,314 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:35,314 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/A, priority=11, startTime=1682326715241; duration=0sec 2023-04-24 08:58:35,314 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:35,314 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:58:35,316 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:58:35,316 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/C is initiating minor compaction (all files) 2023-04-24 08:58:35,316 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/C in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:35,316 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c4e74b41da754b7dbbb399f967c5b0c3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f8370e68f975414c9bae5876dbfa61f0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/e50ab579929a49a7b6f2b5a8cc913618, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/d5015532e53e45c595de6a1da0ee9feb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/904346432a39489cbe9d261e354a3c08] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=60.1 K 2023-04-24 08:58:35,317 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c4e74b41da754b7dbbb399f967c5b0c3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1682326710555 2023-04-24 08:58:35,317 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f8370e68f975414c9bae5876dbfa61f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1682326710870 2023-04-24 08:58:35,317 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e50ab579929a49a7b6f2b5a8cc913618, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1682326711561 2023-04-24 08:58:35,318 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting d5015532e53e45c595de6a1da0ee9feb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1682326712189 2023-04-24 08:58:35,318 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 904346432a39489cbe9d261e354a3c08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682326714335 2023-04-24 08:58:35,333 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#C#compaction#72 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:35,343 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/bd5a6ab7e42946dc9bca29154d67a474 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/bd5a6ab7e42946dc9bca29154d67a474 2023-04-24 08:58:35,349 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a1f74229b819199749846ce3feed93fe/C of a1f74229b819199749846ce3feed93fe into bd5a6ab7e42946dc9bca29154d67a474(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:35,349 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:35,349 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/C, priority=11, startTime=1682326715241; duration=0sec 2023-04-24 08:58:35,349 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:35,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:35,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:35,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:35,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326775611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,693 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/83542661bace463198d83014d675cbe9 2023-04-24 08:58:35,711 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/7904a97d5b864515b666dbc2bff92f39 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7904a97d5b864515b666dbc2bff92f39 2023-04-24 08:58:35,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:35,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326775715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:35,716 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a1f74229b819199749846ce3feed93fe/B of a1f74229b819199749846ce3feed93fe into 7904a97d5b864515b666dbc2bff92f39(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:35,716 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:35,716 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/B, priority=11, startTime=1682326715241; duration=0sec 2023-04-24 08:58:35,716 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:35,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:35,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326775917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,105 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/2270000ebba44533ac2740650d4a58d8 2023-04-24 08:58:36,123 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/6717360ec4104d1a822d02700feda97f 2023-04-24 08:58:36,128 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/83542661bace463198d83014d675cbe9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/83542661bace463198d83014d675cbe9 2023-04-24 08:58:36,133 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/83542661bace463198d83014d675cbe9, entries=150, sequenceid=462, filesize=11.9 K 2023-04-24 08:58:36,135 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/2270000ebba44533ac2740650d4a58d8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2270000ebba44533ac2740650d4a58d8 2023-04-24 08:58:36,140 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2270000ebba44533ac2740650d4a58d8, entries=150, sequenceid=462, filesize=11.9 K 2023-04-24 08:58:36,141 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/6717360ec4104d1a822d02700feda97f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/6717360ec4104d1a822d02700feda97f 2023-04-24 08:58:36,146 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/6717360ec4104d1a822d02700feda97f, entries=150, sequenceid=462, filesize=11.9 K 2023-04-24 08:58:36,147 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for a1f74229b819199749846ce3feed93fe in 883ms, sequenceid=462, compaction requested=false 2023-04-24 08:58:36,147 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:36,147 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:36,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:36,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:36,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:36,183 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23a9f1c1[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:36,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:36,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,191 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,191 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:36,191 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:36,191 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:36,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:36,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,199 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,199 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:36,200 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:36,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:36,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:36,202 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:36,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:36,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:36,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:36,202 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:36,208 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,208 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,208 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,208 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 08:58:36,220 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:36,220 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,221 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:36,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:36,221 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:36,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,224 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,224 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:36,225 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:36,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:36,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:36,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:36,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:36,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:36,225 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:36,225 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:36,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:36,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,225 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,226 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:36,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:36,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326716251, max timestamp=1682326716269Num uniques 859; 2023-04-24 08:58:36,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326716251, max timestamp=1682326716269Num uniques 859; 2023-04-24 08:58:36,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:36,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:36,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326716251, max timestamp=1682326716269Num uniques 859; 2023-04-24 08:58:36,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:36,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:36,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326776320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,325 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:36,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:36,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:36,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:36,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:36,330 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:36,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:36,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:36,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:36,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,340 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:36,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:36,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:36,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:36,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:36,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:36,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:36,342 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:36,342 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:36,342 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:36,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:36,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:36,343 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,343 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:36,343 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:36,343 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:36,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:36,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:36,343 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:36,371 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:36,373 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:36,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@780566d6[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:36,373 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:36,383 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,383 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:36,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:36,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:36,391 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,391 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,391 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,391 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:36,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:36,393 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:36,393 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:36,393 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:36,393 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:36,393 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:36,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,400 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,408 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:36,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:36,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:36,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:36,408 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:36,408 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:36,408 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:36,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:36,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:36,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326776425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,508 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:36,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:36,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:36,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:36,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:36,511 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:36,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:36,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:36,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:36,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,512 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,516 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,516 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:36,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:36,516 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,516 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:36,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:36,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:36,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:36,517 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:36,517 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:36,517 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:36,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:36,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:36,518 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,518 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:36,518 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:36,518 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:36,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:36,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:36,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:36,524 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:36,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:36,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4394ff[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:36,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:36,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:36,533 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,533 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:36,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:36,541 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:36,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,541 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,541 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:36,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:36,543 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:36,543 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:36,543 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:36,543 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:36,543 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:36,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:36,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,550 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,551 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:36,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:36,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:36,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:36,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:36,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,558 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:36,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:36,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:36,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:36,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,558 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,559 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:36,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:36,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326776630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/829cd318b7a043b2b40c07dfa6091497 2023-04-24 08:58:36,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:36,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:36,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:36,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:36,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:36,662 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:36,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:36,663 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:36,663 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:36,663 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,663 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,666 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:36,666 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:36,666 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,666 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:36,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:36,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:36,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:36,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:36,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:36,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:36,675 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:36,675 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,675 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:36,675 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:36,675 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:36,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:36,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:36,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:36,679 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/c815c6c4301141c5b574590d358be7a6 2023-04-24 08:58:36,683 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,683 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,683 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,683 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:36,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3788041d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:36,687 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:36,687 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,691 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:36,691 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:36,691 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:36,691 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:36,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/ed1194704c0847f69866732fd4ff817e 2023-04-24 08:58:36,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,699 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:36,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:36,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:36,703 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:36,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:36,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:36,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:36,703 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:36,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/829cd318b7a043b2b40c07dfa6091497 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/829cd318b7a043b2b40c07dfa6091497 2023-04-24 08:58:36,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:36,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:36,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:36,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,710 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,712 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/829cd318b7a043b2b40c07dfa6091497, entries=150, sequenceid=476, filesize=11.9 K 2023-04-24 08:58:36,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/c815c6c4301141c5b574590d358be7a6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c815c6c4301141c5b574590d358be7a6 2023-04-24 08:58:36,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,716 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:36,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:36,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:36,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:36,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:36,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,716 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:36,717 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:36,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:36,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:36,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:36,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c815c6c4301141c5b574590d358be7a6, entries=150, sequenceid=476, filesize=11.9 K 2023-04-24 08:58:36,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/ed1194704c0847f69866732fd4ff817e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/ed1194704c0847f69866732fd4ff817e 2023-04-24 08:58:36,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/ed1194704c0847f69866732fd4ff817e, entries=150, sequenceid=476, filesize=11.9 K 2023-04-24 08:58:36,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a1f74229b819199749846ce3feed93fe in 507ms, sequenceid=476, compaction requested=true 2023-04-24 08:58:36,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:36,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:36,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:36,727 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:36,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:36,728 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:36,729 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:36,729 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:36,729 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/A is initiating minor compaction (all files) 2023-04-24 08:58:36,729 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/B is initiating minor compaction (all files) 2023-04-24 08:58:36,729 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/A in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,729 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/B in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,729 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4793ea1fc01b4a8a9a0a00adc8a3ba21, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/83542661bace463198d83014d675cbe9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/829cd318b7a043b2b40c07dfa6091497] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=36.5 K 2023-04-24 08:58:36,729 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7904a97d5b864515b666dbc2bff92f39, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2270000ebba44533ac2740650d4a58d8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c815c6c4301141c5b574590d358be7a6] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=36.5 K 2023-04-24 08:58:36,729 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 4793ea1fc01b4a8a9a0a00adc8a3ba21, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682326714335 2023-04-24 08:58:36,729 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7904a97d5b864515b666dbc2bff92f39, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682326714335 2023-04-24 08:58:36,730 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 83542661bace463198d83014d675cbe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682326714452 2023-04-24 08:58:36,730 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 2270000ebba44533ac2740650d4a58d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682326714452 2023-04-24 08:58:36,730 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 829cd318b7a043b2b40c07dfa6091497, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682326715599 2023-04-24 08:58:36,730 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c815c6c4301141c5b574590d358be7a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682326715599 2023-04-24 08:58:36,741 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#B#compaction#78 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:36,745 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#A#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:36,763 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/b67deeb6453242d8a33257340c44d60b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b67deeb6453242d8a33257340c44d60b 2023-04-24 08:58:36,768 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/A of a1f74229b819199749846ce3feed93fe into b67deeb6453242d8a33257340c44d60b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:36,769 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:36,769 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/A, priority=13, startTime=1682326716727; duration=0sec 2023-04-24 08:58:36,769 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:36,769 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:58:36,770 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:58:36,770 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/C is initiating minor compaction (all files) 2023-04-24 08:58:36,770 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/C in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,770 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/bd5a6ab7e42946dc9bca29154d67a474, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/6717360ec4104d1a822d02700feda97f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/ed1194704c0847f69866732fd4ff817e] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=36.5 K 2023-04-24 08:58:36,770 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting bd5a6ab7e42946dc9bca29154d67a474, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1682326714335 2023-04-24 08:58:36,771 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 6717360ec4104d1a822d02700feda97f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682326714452 2023-04-24 08:58:36,771 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting ed1194704c0847f69866732fd4ff817e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682326715599 2023-04-24 08:58:36,777 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#C#compaction#80 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:36,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:36,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:36,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:36,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:36,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:36,820 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:36,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:36,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:36,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:36,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,824 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:36,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:36,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:36,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:36,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:36,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:36,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:36,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:36,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:36,826 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:36,826 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:36,826 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:36,826 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-24 08:58:36,826 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:36,826 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:36,826 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:36,826 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:36,826 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:36,826 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:36,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:36,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:36,838 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/f8f446f9dcbd442797c400fa0930b1c2 2023-04-24 08:58:36,848 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/2e179cffc81f4384bc22c3dc67545180 2023-04-24 08:58:36,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:36,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:36,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:36,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326776956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:37,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326777060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,162 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/3ca99edcf94d4b3a989f1973258c4bda as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/3ca99edcf94d4b3a989f1973258c4bda 2023-04-24 08:58:37,170 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/B of a1f74229b819199749846ce3feed93fe into 3ca99edcf94d4b3a989f1973258c4bda(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:37,170 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:37,171 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/B, priority=13, startTime=1682326716727; duration=0sec 2023-04-24 08:58:37,171 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:37,188 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/12d98f4de7854cffaa293294698ac35b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/12d98f4de7854cffaa293294698ac35b 2023-04-24 08:58:37,193 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a1f74229b819199749846ce3feed93fe/C of a1f74229b819199749846ce3feed93fe into 12d98f4de7854cffaa293294698ac35b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:37,193 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:37,193 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/C, priority=13, startTime=1682326716728; duration=0sec 2023-04-24 08:58:37,193 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:37,260 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/3139805b3967433bae01892329474d84 2023-04-24 08:58:37,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:37,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326777264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,266 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/f8f446f9dcbd442797c400fa0930b1c2 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f8f446f9dcbd442797c400fa0930b1c2 2023-04-24 08:58:37,270 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f8f446f9dcbd442797c400fa0930b1c2, entries=150, sequenceid=513, filesize=11.9 K 2023-04-24 08:58:37,272 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/2e179cffc81f4384bc22c3dc67545180 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2e179cffc81f4384bc22c3dc67545180 2023-04-24 08:58:37,276 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2e179cffc81f4384bc22c3dc67545180, entries=150, sequenceid=513, filesize=11.9 K 2023-04-24 08:58:37,277 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/3139805b3967433bae01892329474d84 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3139805b3967433bae01892329474d84 2023-04-24 08:58:37,282 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3139805b3967433bae01892329474d84, entries=150, sequenceid=513, filesize=11.9 K 2023-04-24 08:58:37,283 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a1f74229b819199749846ce3feed93fe in 457ms, sequenceid=513, compaction requested=false 2023-04-24 08:58:37,283 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:37,283 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:37,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:37,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:37,283 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,284 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:37,284 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:37,291 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:37,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:37,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:37,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:37,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc3ccd4[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:37,292 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:37,293 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:37,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:37,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:37,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:37,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:37,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:37,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:37,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:37,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:37,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:37,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:37,335 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:37,335 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:37,335 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:37,335 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:37,335 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:37,341 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,341 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:37,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,341 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:37,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:37,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:37,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,349 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,349 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:37,349 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:37,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:37,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:37,349 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,350 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:37,350 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:37,350 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,349 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:37,350 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:37,350 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:37,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:37,352 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:37,352 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:37,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:37,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:37,450 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:37,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:37,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:37,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:37,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:37,453 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:37,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:37,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:37,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:37,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:37,458 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:37,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:37,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:37,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:37,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:37,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:37,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:37,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:37,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:37,459 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:37,459 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:37,459 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:37,459 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:37,459 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:37,459 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:37,460 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:37,460 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 08:58:37,460 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:37,460 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:37,460 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:37,460 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:37,460 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:37,460 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:37,467 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/a164132e3add497cb0a7233276c41e85 2023-04-24 08:58:37,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:37,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:37,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326717596, max timestamp=1682326717607Num uniques 859; 2023-04-24 08:58:37,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:37,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326717596, max timestamp=1682326717607Num uniques 859; 2023-04-24 08:58:37,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326717596, max timestamp=1682326717607Num uniques 859; 2023-04-24 08:58:37,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:37,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:37,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:37,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326777641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:37,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326777744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:37,894 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/4b202c9e64294efc80614011bd7c58e2 2023-04-24 08:58:37,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:37,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326777947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:38,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326778250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,306 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/f65cb6ab479a471f828e2cdc3aac3ed9 2023-04-24 08:58:38,315 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/a164132e3add497cb0a7233276c41e85 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/a164132e3add497cb0a7233276c41e85 2023-04-24 08:58:38,322 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/a164132e3add497cb0a7233276c41e85, entries=150, sequenceid=526, filesize=11.9 K 2023-04-24 08:58:38,323 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/4b202c9e64294efc80614011bd7c58e2 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4b202c9e64294efc80614011bd7c58e2 2023-04-24 08:58:38,328 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4b202c9e64294efc80614011bd7c58e2, entries=150, sequenceid=526, filesize=11.9 K 2023-04-24 08:58:38,329 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/f65cb6ab479a471f828e2cdc3aac3ed9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f65cb6ab479a471f828e2cdc3aac3ed9 2023-04-24 08:58:38,334 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f65cb6ab479a471f828e2cdc3aac3ed9, entries=150, sequenceid=526, filesize=11.9 K 2023-04-24 08:58:38,335 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for a1f74229b819199749846ce3feed93fe in 875ms, sequenceid=526, compaction requested=true 2023-04-24 08:58:38,335 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:38,335 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:38,335 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:38,335 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:38,335 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:38,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:38,366 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,366 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:38,366 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:38,366 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:38,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:38,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:38,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:38,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:38,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:38,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8a332ac[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:38,368 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:38,368 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:38,375 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:38,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:38,375 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:38,375 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:38,375 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:38,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:38,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:38,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:38,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:38,383 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:38,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:38,390 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:38,390 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:38,390 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:38,390 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:38,390 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:38,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:38,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:38,400 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:38,404 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:38,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:38,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:38,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:38,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:38,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:38,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:38,416 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:38,416 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:38,416 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:38,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:38,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:38,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:38,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:38,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:38,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:38,516 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:38,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:38,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:38,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:38,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:38,520 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:38,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:38,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:38,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:38,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:38,524 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:38,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:38,525 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,525 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:38,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:38,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:38,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:38,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:38,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:38,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:38,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:38,526 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:38,526 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:38,526 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:38,526 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:38,526 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. started... 2023-04-24 08:58:38,526 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-24 08:58:38,526 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:38,526 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:38,526 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:38,526 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:38,526 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:38,526 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:38,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:38,755 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. as already flushing 2023-04-24 08:58:38,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:38,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326778775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:38,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326778878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:38,935 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/3bb12253f8df498c9329f84188580cff 2023-04-24 08:58:39,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:39,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326779081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,315 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13065328 to 127.0.0.1:63978 2023-04-24 08:58:39,316 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:39,316 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a6a210a to 127.0.0.1:63978 2023-04-24 08:58:39,316 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:39,318 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7abbc496 to 127.0.0.1:63978 2023-04-24 08:58:39,318 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:39,319 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11a028ee to 127.0.0.1:63978 2023-04-24 08:58:39,319 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:39,347 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/7f8873494a79438f9e8a78f185a90c9e 2023-04-24 08:58:39,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:39,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57328 deadline: 1682326779387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,765 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/8d8190412e6f4790be7ffde16a7d2853 2023-04-24 08:58:39,776 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/3bb12253f8df498c9329f84188580cff as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/3bb12253f8df498c9329f84188580cff 2023-04-24 08:58:39,781 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/3bb12253f8df498c9329f84188580cff, entries=150, sequenceid=561, filesize=11.9 K 2023-04-24 08:58:39,782 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/7f8873494a79438f9e8a78f185a90c9e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7f8873494a79438f9e8a78f185a90c9e 2023-04-24 08:58:39,786 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7f8873494a79438f9e8a78f185a90c9e, entries=150, sequenceid=561, filesize=11.9 K 2023-04-24 08:58:39,787 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/8d8190412e6f4790be7ffde16a7d2853 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8d8190412e6f4790be7ffde16a7d2853 2023-04-24 08:58:39,791 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8d8190412e6f4790be7ffde16a7d2853, entries=150, sequenceid=561, filesize=11.9 K 2023-04-24 08:58:39,792 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for a1f74229b819199749846ce3feed93fe in 1266ms, sequenceid=561, compaction requested=true 2023-04-24 08:58:39,792 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:39,792 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:39,792 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:39,792 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:39,792 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,792 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:39,792 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:39,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,825 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:39,825 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:39,825 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:39,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:39,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b14ab2b[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:39,829 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:39,829 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:39,833 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:39,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:39,833 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:39,833 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:39,833 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:39,833 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,833 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:39,841 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:39,841 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:39,841 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:39,841 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:39,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:39,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:39,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:39,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:39,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:39,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:39,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:39,850 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:39,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:39,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:39,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:39,850 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:39,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:39,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:39,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:39,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:39,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:39,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:39,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:39,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:39,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:39,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:39,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:39,859 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:39,862 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,874 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:39,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:39,874 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:39,875 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:39,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:39,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:39,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:39,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:39,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:39,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:39,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:39,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:39,875 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:39,875 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:39,876 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:39,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:39,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:39,894 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 08:58:39,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:39,895 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7de00027 to 127.0.0.1:63978 2023-04-24 08:58:39,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:39,895 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:39,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:39,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:39,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:39,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:39,976 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:39,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:40,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/b1c709b1f0bb480585507667f724e3ea 2023-04-24 08:58:40,724 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/b78ef13ab09e461cba9a74763620b284 2023-04-24 08:58:41,139 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/4bef660a7a0f494c9e7c07fc176e2e9e 2023-04-24 08:58:41,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/b1c709b1f0bb480585507667f724e3ea as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b1c709b1f0bb480585507667f724e3ea 2023-04-24 08:58:41,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b1c709b1f0bb480585507667f724e3ea, entries=150, sequenceid=573, filesize=11.9 K 2023-04-24 08:58:41,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/b78ef13ab09e461cba9a74763620b284 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/b78ef13ab09e461cba9a74763620b284 2023-04-24 08:58:41,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/b78ef13ab09e461cba9a74763620b284, entries=150, sequenceid=573, filesize=11.9 K 2023-04-24 08:58:41,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/4bef660a7a0f494c9e7c07fc176e2e9e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4bef660a7a0f494c9e7c07fc176e2e9e 2023-04-24 08:58:41,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4bef660a7a0f494c9e7c07fc176e2e9e, entries=150, sequenceid=573, filesize=11.9 K 2023-04-24 08:58:41,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for a1f74229b819199749846ce3feed93fe in 1270ms, sequenceid=573, compaction requested=true 2023-04-24 08:58:41,165 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:41,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:41,165 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:58:41,165 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:58:41,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:41,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:41,167 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:58:41,167 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:58:41,167 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/A is initiating minor compaction (all files) 2023-04-24 08:58:41,167 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/B is initiating minor compaction (all files) 2023-04-24 08:58:41,167 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/A in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:41,167 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/B in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:41,167 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b67deeb6453242d8a33257340c44d60b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f8f446f9dcbd442797c400fa0930b1c2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/a164132e3add497cb0a7233276c41e85, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/3bb12253f8df498c9329f84188580cff, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b1c709b1f0bb480585507667f724e3ea] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=60.4 K 2023-04-24 08:58:41,167 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/3ca99edcf94d4b3a989f1973258c4bda, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2e179cffc81f4384bc22c3dc67545180, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4b202c9e64294efc80614011bd7c58e2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7f8873494a79438f9e8a78f185a90c9e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/b78ef13ab09e461cba9a74763620b284] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=60.4 K 2023-04-24 08:58:41,167 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting b67deeb6453242d8a33257340c44d60b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682326715599 2023-04-24 08:58:41,168 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3ca99edcf94d4b3a989f1973258c4bda, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682326715599 2023-04-24 08:58:41,168 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f8f446f9dcbd442797c400fa0930b1c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1682326716311 2023-04-24 08:58:41,168 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 2e179cffc81f4384bc22c3dc67545180, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1682326716311 2023-04-24 08:58:41,168 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting a164132e3add497cb0a7233276c41e85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682326716941 2023-04-24 08:58:41,168 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4b202c9e64294efc80614011bd7c58e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682326716941 2023-04-24 08:58:41,169 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 3bb12253f8df498c9329f84188580cff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1682326717626 2023-04-24 08:58:41,169 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7f8873494a79438f9e8a78f185a90c9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1682326717626 2023-04-24 08:58:41,169 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting b1c709b1f0bb480585507667f724e3ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682326718761 2023-04-24 08:58:41,169 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting b78ef13ab09e461cba9a74763620b284, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682326718761 2023-04-24 08:58:41,181 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#A#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:41,184 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#B#compaction#94 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:41,599 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/0a93377c8a434df5b3a542041b820596 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0a93377c8a434df5b3a542041b820596 2023-04-24 08:58:41,599 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/c10c6ea9a9624987864eb7c7182dfb0f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c10c6ea9a9624987864eb7c7182dfb0f 2023-04-24 08:58:41,605 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a1f74229b819199749846ce3feed93fe/B of a1f74229b819199749846ce3feed93fe into c10c6ea9a9624987864eb7c7182dfb0f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:41,605 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a1f74229b819199749846ce3feed93fe/A of a1f74229b819199749846ce3feed93fe into 0a93377c8a434df5b3a542041b820596(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:41,605 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:41,605 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:41,605 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/B, priority=11, startTime=1682326721165; duration=0sec 2023-04-24 08:58:41,605 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/A, priority=11, startTime=1682326721165; duration=0sec 2023-04-24 08:58:41,605 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:41,605 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:41,605 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:58:41,607 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:58:41,607 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): a1f74229b819199749846ce3feed93fe/C is initiating minor compaction (all files) 2023-04-24 08:58:41,608 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a1f74229b819199749846ce3feed93fe/C in TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:41,608 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/12d98f4de7854cffaa293294698ac35b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3139805b3967433bae01892329474d84, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f65cb6ab479a471f828e2cdc3aac3ed9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8d8190412e6f4790be7ffde16a7d2853, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4bef660a7a0f494c9e7c07fc176e2e9e] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp, totalSize=60.4 K 2023-04-24 08:58:41,608 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 12d98f4de7854cffaa293294698ac35b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1682326715599 2023-04-24 08:58:41,609 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3139805b3967433bae01892329474d84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=513, earliestPutTs=1682326716311 2023-04-24 08:58:41,609 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f65cb6ab479a471f828e2cdc3aac3ed9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1682326716941 2023-04-24 08:58:41,609 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 8d8190412e6f4790be7ffde16a7d2853, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1682326717626 2023-04-24 08:58:41,610 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4bef660a7a0f494c9e7c07fc176e2e9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1682326718761 2023-04-24 08:58:41,621 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): a1f74229b819199749846ce3feed93fe#C#compaction#95 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:42,040 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/271b0fc2a2df4553b5080f1d3c85d108 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/271b0fc2a2df4553b5080f1d3c85d108 2023-04-24 08:58:42,047 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a1f74229b819199749846ce3feed93fe/C of a1f74229b819199749846ce3feed93fe into 271b0fc2a2df4553b5080f1d3c85d108(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:42,047 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:42,047 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe., storeName=a1f74229b819199749846ce3feed93fe/C, priority=11, startTime=1682326721165; duration=0sec 2023-04-24 08:58:42,048 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:43,039 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6194be61 to 127.0.0.1:63978 2023-04-24 08:58:43,040 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:43,060 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0446e24c to 127.0.0.1:63978 2023-04-24 08:58:43,060 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:44,892 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d114eb9 to 127.0.0.1:63978 2023-04-24 08:58:44,892 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:44,900 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04d5a66f to 127.0.0.1:63978 2023-04-24 08:58:44,900 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:44,900 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 08:58:44,901 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 62 2023-04-24 08:58:44,901 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-24 08:58:44,901 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-04-24 08:58:44,901 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 274 2023-04-24 08:58:44,901 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-04-24 08:58:44,901 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 08:58:44,902 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5316 2023-04-24 08:58:44,902 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5424 2023-04-24 08:58:44,902 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 08:58:44,902 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2342 2023-04-24 08:58:44,902 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7023 rows 2023-04-24 08:58:44,902 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2342 2023-04-24 08:58:44,902 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7023 rows 2023-04-24 08:58:44,902 INFO [Listener at localhost.localdomain/44291] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 08:58:44,902 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x482e2663 to 127.0.0.1:63978 2023-04-24 08:58:44,903 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:58:44,910 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 08:58:44,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-24 08:58:44,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:44,922 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326724922"}]},"ts":"1682326724922"} 2023-04-24 08:58:44,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 08:58:44,923 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 08:58:44,949 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 08:58:44,951 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a1f74229b819199749846ce3feed93fe, UNASSIGN}] 2023-04-24 08:58:44,954 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a1f74229b819199749846ce3feed93fe, UNASSIGN 2023-04-24 08:58:44,955 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a1f74229b819199749846ce3feed93fe, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:44,955 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326724955"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326724955"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326724955"}]},"ts":"1682326724955"} 2023-04-24 08:58:44,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:58:45,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 08:58:45,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:45,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing a1f74229b819199749846ce3feed93fe, disabling compactions & flushes 2023-04-24 08:58:45,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:45,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:45,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. after waiting 0 ms 2023-04-24 08:58:45,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:45,121 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing a1f74229b819199749846ce3feed93fe 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 08:58:45,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=A 2023-04-24 08:58:45,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:45,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=B 2023-04-24 08:58:45,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:45,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a1f74229b819199749846ce3feed93fe, store=C 2023-04-24 08:58:45,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:45,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 08:58:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 08:58:45,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/b249825d6440443685defb49935e382a 2023-04-24 08:58:45,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/606fa45ed3444f16bd6c613affee702c 2023-04-24 08:58:46,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 08:58:46,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/dfb8cc50879e4341968455fc942aab0b 2023-04-24 08:58:46,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/A/b249825d6440443685defb49935e382a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b249825d6440443685defb49935e382a 2023-04-24 08:58:46,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b249825d6440443685defb49935e382a, entries=150, sequenceid=583, filesize=11.9 K 2023-04-24 08:58:46,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/B/606fa45ed3444f16bd6c613affee702c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/606fa45ed3444f16bd6c613affee702c 2023-04-24 08:58:46,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/606fa45ed3444f16bd6c613affee702c, entries=150, sequenceid=583, filesize=11.9 K 2023-04-24 08:58:46,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/.tmp/C/dfb8cc50879e4341968455fc942aab0b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/dfb8cc50879e4341968455fc942aab0b 2023-04-24 08:58:46,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/dfb8cc50879e4341968455fc942aab0b, entries=150, sequenceid=583, filesize=11.9 K 2023-04-24 08:58:46,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for a1f74229b819199749846ce3feed93fe in 1291ms, sequenceid=583, compaction requested=false 2023-04-24 08:58:46,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/6987e13f212045d897e3a9a06fa084fc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/55f28a36e7c24c9682f79c45721fdc4b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9020a8b57b3a4396941f0b95412410d8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5969ff67bbd9421897da489a5e0c5270, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/51255a66b74d4479993999049494af02, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/2b67e509c97f425bbc0f21a0c2f30542, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/d09093964f7041a78bbd0c7bba277461, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5b8b695c4c544c67b42a1dd8323ae676, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/948dd94e78524c8da4d49f0b5841e01e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f7f1a2851eae4ccb9c27ba4566060fe7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f6248cd3ea1d40bf841a463b46ac18b9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/bc85857fbf544118a2deade60507f302, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9f3e70d4d0ec462fbc1b597c3459e533, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/532a2bab2fe843e0874807e328102fb0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4284046c64db4bec9cd9cc2aac563d65, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b6249aca37534396a1213a9a5e4dd699, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/c949cba834bd41159a12fa877dfeca70, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/76535ca8c0c347db9d53a0470613fdd9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0e8e4a8ca8e74205b645b1db9314b0c5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/ab6a456ba68e439c9bf51fa07aa46de2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/028a4d48a453446785418c5d1c07114f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0844b7ecbe3442f0974cb82f39d8e62e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/536ffb91ffa84c57a46f71bc64f8baa4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4793ea1fc01b4a8a9a0a00adc8a3ba21, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/83542661bace463198d83014d675cbe9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b67deeb6453242d8a33257340c44d60b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/829cd318b7a043b2b40c07dfa6091497, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f8f446f9dcbd442797c400fa0930b1c2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/a164132e3add497cb0a7233276c41e85, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/3bb12253f8df498c9329f84188580cff, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b1c709b1f0bb480585507667f724e3ea] to archive 2023-04-24 08:58:46,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:58:46,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/6987e13f212045d897e3a9a06fa084fc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/6987e13f212045d897e3a9a06fa084fc 2023-04-24 08:58:46,439 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/55f28a36e7c24c9682f79c45721fdc4b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/55f28a36e7c24c9682f79c45721fdc4b 2023-04-24 08:58:46,440 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9020a8b57b3a4396941f0b95412410d8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9020a8b57b3a4396941f0b95412410d8 2023-04-24 08:58:46,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5969ff67bbd9421897da489a5e0c5270 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5969ff67bbd9421897da489a5e0c5270 2023-04-24 08:58:46,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/51255a66b74d4479993999049494af02 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/51255a66b74d4479993999049494af02 2023-04-24 08:58:46,445 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/2b67e509c97f425bbc0f21a0c2f30542 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/2b67e509c97f425bbc0f21a0c2f30542 2023-04-24 08:58:46,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/d09093964f7041a78bbd0c7bba277461 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/d09093964f7041a78bbd0c7bba277461 2023-04-24 08:58:46,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5b8b695c4c544c67b42a1dd8323ae676 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/5b8b695c4c544c67b42a1dd8323ae676 2023-04-24 08:58:46,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/948dd94e78524c8da4d49f0b5841e01e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/948dd94e78524c8da4d49f0b5841e01e 2023-04-24 08:58:46,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f7f1a2851eae4ccb9c27ba4566060fe7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f7f1a2851eae4ccb9c27ba4566060fe7 2023-04-24 08:58:46,452 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f6248cd3ea1d40bf841a463b46ac18b9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f6248cd3ea1d40bf841a463b46ac18b9 2023-04-24 08:58:46,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/bc85857fbf544118a2deade60507f302 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/bc85857fbf544118a2deade60507f302 2023-04-24 08:58:46,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9f3e70d4d0ec462fbc1b597c3459e533 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/9f3e70d4d0ec462fbc1b597c3459e533 2023-04-24 08:58:46,457 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/532a2bab2fe843e0874807e328102fb0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/532a2bab2fe843e0874807e328102fb0 2023-04-24 08:58:46,459 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4284046c64db4bec9cd9cc2aac563d65 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4284046c64db4bec9cd9cc2aac563d65 2023-04-24 08:58:46,461 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b6249aca37534396a1213a9a5e4dd699 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b6249aca37534396a1213a9a5e4dd699 2023-04-24 08:58:46,463 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/c949cba834bd41159a12fa877dfeca70 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/c949cba834bd41159a12fa877dfeca70 2023-04-24 08:58:46,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/76535ca8c0c347db9d53a0470613fdd9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/76535ca8c0c347db9d53a0470613fdd9 2023-04-24 08:58:46,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0e8e4a8ca8e74205b645b1db9314b0c5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0e8e4a8ca8e74205b645b1db9314b0c5 2023-04-24 08:58:46,467 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/ab6a456ba68e439c9bf51fa07aa46de2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/ab6a456ba68e439c9bf51fa07aa46de2 2023-04-24 08:58:46,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/028a4d48a453446785418c5d1c07114f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/028a4d48a453446785418c5d1c07114f 2023-04-24 08:58:46,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0844b7ecbe3442f0974cb82f39d8e62e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0844b7ecbe3442f0974cb82f39d8e62e 2023-04-24 08:58:46,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/536ffb91ffa84c57a46f71bc64f8baa4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/536ffb91ffa84c57a46f71bc64f8baa4 2023-04-24 08:58:46,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4793ea1fc01b4a8a9a0a00adc8a3ba21 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/4793ea1fc01b4a8a9a0a00adc8a3ba21 2023-04-24 08:58:46,472 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/83542661bace463198d83014d675cbe9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/83542661bace463198d83014d675cbe9 2023-04-24 08:58:46,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b67deeb6453242d8a33257340c44d60b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b67deeb6453242d8a33257340c44d60b 2023-04-24 08:58:46,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/829cd318b7a043b2b40c07dfa6091497 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/829cd318b7a043b2b40c07dfa6091497 2023-04-24 08:58:46,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f8f446f9dcbd442797c400fa0930b1c2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/f8f446f9dcbd442797c400fa0930b1c2 2023-04-24 08:58:46,475 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/a164132e3add497cb0a7233276c41e85 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/a164132e3add497cb0a7233276c41e85 2023-04-24 08:58:46,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/3bb12253f8df498c9329f84188580cff to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/3bb12253f8df498c9329f84188580cff 2023-04-24 08:58:46,477 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b1c709b1f0bb480585507667f724e3ea to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b1c709b1f0bb480585507667f724e3ea 2023-04-24 08:58:46,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4fdb3c0b5fbe44acb73fe727073003f1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c1dca11619574eb283333edad3a2ece1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/44d8002a1fff44e8b6dda2a6860c92e6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/d57a3407fd3f4b96b6bde3d2bdd11d12, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8261d3d6cc9d430bbb9fa2dc44bfd520, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4c1d6acf22b04e43a87fe396cdbb69c7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6fa81489bda647a888fda5ea72aefa03, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c50caec43fdc47d386f61e9b2c23020d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/38f33704f6484b6381ae17dfcbbb7529, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c98530b9681848f38ea06d31d94fa0f7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f8eb7f1e8cd14e808e91fb98b66761de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6a530dca3e9540559ea821db5c857a9c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/9ee8e6b4c2d14a63849ab15ed862c24a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f275bc14c6a94b9d9721072339ed3b75, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8d85db58e49c41d7af6aa4f57687124d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/415e283334d84f1fa41f3438dc6bfdf1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/64ce7026219f48efb05afaf74ea70579, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/e6b2e511c20346e29ec94247d909b3bb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/a9c284ea6436432392cc0c693d4cc1a7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7467f51ebf5b4fdb99004b6fe3cc6208, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/74e7dfe1f6454f9d87f0e7788210cdf7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/164817d197cd43a48f032d6ebcbc8cea, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7904a97d5b864515b666dbc2bff92f39, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/50ac784f44754e2997c9b2e01667ebdf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2270000ebba44533ac2740650d4a58d8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/3ca99edcf94d4b3a989f1973258c4bda, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c815c6c4301141c5b574590d358be7a6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2e179cffc81f4384bc22c3dc67545180, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4b202c9e64294efc80614011bd7c58e2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7f8873494a79438f9e8a78f185a90c9e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/b78ef13ab09e461cba9a74763620b284] to archive 2023-04-24 08:58:46,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:58:46,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4fdb3c0b5fbe44acb73fe727073003f1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4fdb3c0b5fbe44acb73fe727073003f1 2023-04-24 08:58:46,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c1dca11619574eb283333edad3a2ece1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c1dca11619574eb283333edad3a2ece1 2023-04-24 08:58:46,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/44d8002a1fff44e8b6dda2a6860c92e6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/44d8002a1fff44e8b6dda2a6860c92e6 2023-04-24 08:58:46,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/d57a3407fd3f4b96b6bde3d2bdd11d12 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/d57a3407fd3f4b96b6bde3d2bdd11d12 2023-04-24 08:58:46,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8261d3d6cc9d430bbb9fa2dc44bfd520 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8261d3d6cc9d430bbb9fa2dc44bfd520 2023-04-24 08:58:46,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4c1d6acf22b04e43a87fe396cdbb69c7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4c1d6acf22b04e43a87fe396cdbb69c7 2023-04-24 08:58:46,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6fa81489bda647a888fda5ea72aefa03 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6fa81489bda647a888fda5ea72aefa03 2023-04-24 08:58:46,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c50caec43fdc47d386f61e9b2c23020d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c50caec43fdc47d386f61e9b2c23020d 2023-04-24 08:58:46,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/38f33704f6484b6381ae17dfcbbb7529 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/38f33704f6484b6381ae17dfcbbb7529 2023-04-24 08:58:46,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c98530b9681848f38ea06d31d94fa0f7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c98530b9681848f38ea06d31d94fa0f7 2023-04-24 08:58:46,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f8eb7f1e8cd14e808e91fb98b66761de to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f8eb7f1e8cd14e808e91fb98b66761de 2023-04-24 08:58:46,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6a530dca3e9540559ea821db5c857a9c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/6a530dca3e9540559ea821db5c857a9c 2023-04-24 08:58:46,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/9ee8e6b4c2d14a63849ab15ed862c24a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/9ee8e6b4c2d14a63849ab15ed862c24a 2023-04-24 08:58:46,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f275bc14c6a94b9d9721072339ed3b75 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/f275bc14c6a94b9d9721072339ed3b75 2023-04-24 08:58:46,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8d85db58e49c41d7af6aa4f57687124d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/8d85db58e49c41d7af6aa4f57687124d 2023-04-24 08:58:46,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/415e283334d84f1fa41f3438dc6bfdf1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/415e283334d84f1fa41f3438dc6bfdf1 2023-04-24 08:58:46,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/64ce7026219f48efb05afaf74ea70579 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/64ce7026219f48efb05afaf74ea70579 2023-04-24 08:58:46,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/e6b2e511c20346e29ec94247d909b3bb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/e6b2e511c20346e29ec94247d909b3bb 2023-04-24 08:58:46,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/a9c284ea6436432392cc0c693d4cc1a7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/a9c284ea6436432392cc0c693d4cc1a7 2023-04-24 08:58:46,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7467f51ebf5b4fdb99004b6fe3cc6208 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7467f51ebf5b4fdb99004b6fe3cc6208 2023-04-24 08:58:46,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/74e7dfe1f6454f9d87f0e7788210cdf7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/74e7dfe1f6454f9d87f0e7788210cdf7 2023-04-24 08:58:46,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/164817d197cd43a48f032d6ebcbc8cea to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/164817d197cd43a48f032d6ebcbc8cea 2023-04-24 08:58:46,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7904a97d5b864515b666dbc2bff92f39 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7904a97d5b864515b666dbc2bff92f39 2023-04-24 08:58:46,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/50ac784f44754e2997c9b2e01667ebdf to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/50ac784f44754e2997c9b2e01667ebdf 2023-04-24 08:58:46,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2270000ebba44533ac2740650d4a58d8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2270000ebba44533ac2740650d4a58d8 2023-04-24 08:58:46,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/3ca99edcf94d4b3a989f1973258c4bda to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/3ca99edcf94d4b3a989f1973258c4bda 2023-04-24 08:58:46,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c815c6c4301141c5b574590d358be7a6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c815c6c4301141c5b574590d358be7a6 2023-04-24 08:58:46,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2e179cffc81f4384bc22c3dc67545180 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/2e179cffc81f4384bc22c3dc67545180 2023-04-24 08:58:46,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4b202c9e64294efc80614011bd7c58e2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/4b202c9e64294efc80614011bd7c58e2 2023-04-24 08:58:46,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7f8873494a79438f9e8a78f185a90c9e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/7f8873494a79438f9e8a78f185a90c9e 2023-04-24 08:58:46,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/b78ef13ab09e461cba9a74763620b284 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/b78ef13ab09e461cba9a74763620b284 2023-04-24 08:58:46,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4ea874bce4a34867b606c7c56bf7c32c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8591c0afc367445eb27359876fd902ae, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f9704429a62e4c3381de8172c8d01a23, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/2a8f5501344243a39c2c45fea7aea6ff, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/255e8673985644a1a8eff570d6112075, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1f5ff89df86741da8fa4e3435be71359, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/009863d17dfc4018a95a361606e63cce, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/244172ef37b24a23babc3be750bdc3c0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3c90d70e96bd4a9ab83ade5f8c35cf8f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1a9558441a9b4be48ede79ec0c9ada96, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/5126036c7c6d4b60b330e751c82ee352, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/16148ab5615046e0a72def21c93f49ae, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/76fef896c21d47e490ea50e31d1339bc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3e5a0b16f077452a8c831d18a19e0776, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/14038f9e20fb4481a33636e00c9d2a03, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1851648009ba478691a9c9bb6f845382, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c42e22ac426841079430be2380e5e5d1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c4e74b41da754b7dbbb399f967c5b0c3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/9fe22a630c114c0b9486dd83c2f1c135, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f8370e68f975414c9bae5876dbfa61f0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/e50ab579929a49a7b6f2b5a8cc913618, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/d5015532e53e45c595de6a1da0ee9feb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/bd5a6ab7e42946dc9bca29154d67a474, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/904346432a39489cbe9d261e354a3c08, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/6717360ec4104d1a822d02700feda97f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/12d98f4de7854cffaa293294698ac35b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/ed1194704c0847f69866732fd4ff817e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3139805b3967433bae01892329474d84, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f65cb6ab479a471f828e2cdc3aac3ed9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8d8190412e6f4790be7ffde16a7d2853, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4bef660a7a0f494c9e7c07fc176e2e9e] to archive 2023-04-24 08:58:46,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:58:46,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4ea874bce4a34867b606c7c56bf7c32c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4ea874bce4a34867b606c7c56bf7c32c 2023-04-24 08:58:46,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8591c0afc367445eb27359876fd902ae to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8591c0afc367445eb27359876fd902ae 2023-04-24 08:58:46,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f9704429a62e4c3381de8172c8d01a23 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f9704429a62e4c3381de8172c8d01a23 2023-04-24 08:58:46,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/2a8f5501344243a39c2c45fea7aea6ff to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/2a8f5501344243a39c2c45fea7aea6ff 2023-04-24 08:58:46,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/255e8673985644a1a8eff570d6112075 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/255e8673985644a1a8eff570d6112075 2023-04-24 08:58:46,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1f5ff89df86741da8fa4e3435be71359 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1f5ff89df86741da8fa4e3435be71359 2023-04-24 08:58:46,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/009863d17dfc4018a95a361606e63cce to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/009863d17dfc4018a95a361606e63cce 2023-04-24 08:58:46,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/244172ef37b24a23babc3be750bdc3c0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/244172ef37b24a23babc3be750bdc3c0 2023-04-24 08:58:46,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3c90d70e96bd4a9ab83ade5f8c35cf8f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3c90d70e96bd4a9ab83ade5f8c35cf8f 2023-04-24 08:58:46,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1a9558441a9b4be48ede79ec0c9ada96 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1a9558441a9b4be48ede79ec0c9ada96 2023-04-24 08:58:46,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/5126036c7c6d4b60b330e751c82ee352 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/5126036c7c6d4b60b330e751c82ee352 2023-04-24 08:58:46,557 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/16148ab5615046e0a72def21c93f49ae to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/16148ab5615046e0a72def21c93f49ae 2023-04-24 08:58:46,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/76fef896c21d47e490ea50e31d1339bc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/76fef896c21d47e490ea50e31d1339bc 2023-04-24 08:58:46,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3e5a0b16f077452a8c831d18a19e0776 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3e5a0b16f077452a8c831d18a19e0776 2023-04-24 08:58:46,560 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/14038f9e20fb4481a33636e00c9d2a03 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/14038f9e20fb4481a33636e00c9d2a03 2023-04-24 08:58:46,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1851648009ba478691a9c9bb6f845382 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/1851648009ba478691a9c9bb6f845382 2023-04-24 08:58:46,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c42e22ac426841079430be2380e5e5d1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c42e22ac426841079430be2380e5e5d1 2023-04-24 08:58:46,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c4e74b41da754b7dbbb399f967c5b0c3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/c4e74b41da754b7dbbb399f967c5b0c3 2023-04-24 08:58:46,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/9fe22a630c114c0b9486dd83c2f1c135 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/9fe22a630c114c0b9486dd83c2f1c135 2023-04-24 08:58:46,566 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f8370e68f975414c9bae5876dbfa61f0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f8370e68f975414c9bae5876dbfa61f0 2023-04-24 08:58:46,567 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/e50ab579929a49a7b6f2b5a8cc913618 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/e50ab579929a49a7b6f2b5a8cc913618 2023-04-24 08:58:46,568 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/d5015532e53e45c595de6a1da0ee9feb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/d5015532e53e45c595de6a1da0ee9feb 2023-04-24 08:58:46,569 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/bd5a6ab7e42946dc9bca29154d67a474 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/bd5a6ab7e42946dc9bca29154d67a474 2023-04-24 08:58:46,570 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/904346432a39489cbe9d261e354a3c08 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/904346432a39489cbe9d261e354a3c08 2023-04-24 08:58:46,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/6717360ec4104d1a822d02700feda97f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/6717360ec4104d1a822d02700feda97f 2023-04-24 08:58:46,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/12d98f4de7854cffaa293294698ac35b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/12d98f4de7854cffaa293294698ac35b 2023-04-24 08:58:46,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/ed1194704c0847f69866732fd4ff817e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/ed1194704c0847f69866732fd4ff817e 2023-04-24 08:58:46,574 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3139805b3967433bae01892329474d84 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/3139805b3967433bae01892329474d84 2023-04-24 08:58:46,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f65cb6ab479a471f828e2cdc3aac3ed9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/f65cb6ab479a471f828e2cdc3aac3ed9 2023-04-24 08:58:46,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8d8190412e6f4790be7ffde16a7d2853 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/8d8190412e6f4790be7ffde16a7d2853 2023-04-24 08:58:46,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4bef660a7a0f494c9e7c07fc176e2e9e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/4bef660a7a0f494c9e7c07fc176e2e9e 2023-04-24 08:58:46,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/recovered.edits/586.seqid, newMaxSeqId=586, maxSeqId=1 2023-04-24 08:58:46,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe. 2023-04-24 08:58:46,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for a1f74229b819199749846ce3feed93fe: 2023-04-24 08:58:46,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:46,584 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a1f74229b819199749846ce3feed93fe, regionState=CLOSED 2023-04-24 08:58:46,585 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326726584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326726584"}]},"ts":"1682326726584"} 2023-04-24 08:58:46,588 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-24 08:58:46,588 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure a1f74229b819199749846ce3feed93fe, server=jenkins-hbase16.apache.org,42837,1682326693550 in 1.6280 sec 2023-04-24 08:58:46,589 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-24 08:58:46,590 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a1f74229b819199749846ce3feed93fe, UNASSIGN in 1.6370 sec 2023-04-24 08:58:46,590 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326726590"}]},"ts":"1682326726590"} 2023-04-24 08:58:46,591 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 08:58:46,641 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 08:58:46,646 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7280 sec 2023-04-24 08:58:47,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-24 08:58:47,030 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-24 08:58:47,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-24 08:58:47,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:47,047 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:47,049 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:47,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-24 08:58:47,053 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:47,056 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/recovered.edits] 2023-04-24 08:58:47,060 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0a93377c8a434df5b3a542041b820596 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/0a93377c8a434df5b3a542041b820596 2023-04-24 08:58:47,062 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b249825d6440443685defb49935e382a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/A/b249825d6440443685defb49935e382a 2023-04-24 08:58:47,064 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/606fa45ed3444f16bd6c613affee702c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/606fa45ed3444f16bd6c613affee702c 2023-04-24 08:58:47,065 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c10c6ea9a9624987864eb7c7182dfb0f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/B/c10c6ea9a9624987864eb7c7182dfb0f 2023-04-24 08:58:47,067 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/271b0fc2a2df4553b5080f1d3c85d108 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/271b0fc2a2df4553b5080f1d3c85d108 2023-04-24 08:58:47,068 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/dfb8cc50879e4341968455fc942aab0b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/C/dfb8cc50879e4341968455fc942aab0b 2023-04-24 08:58:47,071 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/recovered.edits/586.seqid to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe/recovered.edits/586.seqid 2023-04-24 08:58:47,071 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/a1f74229b819199749846ce3feed93fe 2023-04-24 08:58:47,071 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 08:58:47,073 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:47,091 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 08:58:47,129 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 08:58:47,130 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:47,130 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 08:58:47,130 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682326727130"}]},"ts":"9223372036854775807"} 2023-04-24 08:58:47,132 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 08:58:47,132 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a1f74229b819199749846ce3feed93fe, NAME => 'TestAcidGuarantees,,1682326697998.a1f74229b819199749846ce3feed93fe.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 08:58:47,132 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 08:58:47,133 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682326727133"}]},"ts":"9223372036854775807"} 2023-04-24 08:58:47,134 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 08:58:47,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-24 08:58:47,167 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:47,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 128 msec 2023-04-24 08:58:47,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-24 08:58:47,355 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-24 08:58:47,381 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=232 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7635eff-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7635eff-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1262272855_15 at /127.0.0.1:53462 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7635eff-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase16:42837-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7635eff-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=560 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=345 (was 225) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=8605 (was 9127) 2023-04-24 08:58:47,391 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=345, ProcessCount=170, AvailableMemoryMB=8604 2023-04-24 08:58:47,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 08:58:47,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 08:58:47,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:47,396 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 08:58:47,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-24 08:58:47,397 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 08:58:47,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 08:58:47,399 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:47,400 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 empty. 2023-04-24 08:58:47,400 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:47,400 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 08:58:47,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 08:58:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 08:58:47,815 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 08:58:47,820 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 415f076aa60cbee3cb48b662f1bba024, NAME => 'TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp 2023-04-24 08:58:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 08:58:48,232 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:48,232 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 415f076aa60cbee3cb48b662f1bba024, disabling compactions & flushes 2023-04-24 08:58:48,232 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:48,233 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:48,233 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. after waiting 0 ms 2023-04-24 08:58:48,233 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:48,233 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:48,233 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:48,239 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 08:58:48,241 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326728240"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326728240"}]},"ts":"1682326728240"} 2023-04-24 08:58:48,242 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 08:58:48,243 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 08:58:48,244 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326728244"}]},"ts":"1682326728244"} 2023-04-24 08:58:48,245 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 08:58:48,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, ASSIGN}] 2023-04-24 08:58:48,295 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, ASSIGN 2023-04-24 08:58:48,296 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=false 2023-04-24 08:58:48,449 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=415f076aa60cbee3cb48b662f1bba024, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:48,449 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326728448"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326728448"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326728448"}]},"ts":"1682326728448"} 2023-04-24 08:58:48,453 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:58:48,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 08:58:48,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:48,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 415f076aa60cbee3cb48b662f1bba024, NAME => 'TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:58:48,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:48,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,622 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,624 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A 2023-04-24 08:58:48,624 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A 2023-04-24 08:58:48,624 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:48,625 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 415f076aa60cbee3cb48b662f1bba024 columnFamilyName A 2023-04-24 08:58:48,625 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(310): Store=415f076aa60cbee3cb48b662f1bba024/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:48,626 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,627 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B 2023-04-24 08:58:48,627 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B 2023-04-24 08:58:48,628 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:48,628 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 415f076aa60cbee3cb48b662f1bba024 columnFamilyName B 2023-04-24 08:58:48,628 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(310): Store=415f076aa60cbee3cb48b662f1bba024/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:48,629 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,630 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C 2023-04-24 08:58:48,630 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C 2023-04-24 08:58:48,630 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:48,631 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 415f076aa60cbee3cb48b662f1bba024 columnFamilyName C 2023-04-24 08:58:48,631 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(310): Store=415f076aa60cbee3cb48b662f1bba024/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:48,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:48,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 08:58:48,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:48,639 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 08:58:48,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 415f076aa60cbee3cb48b662f1bba024; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73307771, jitterRate=0.0923709124326706}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 08:58:48,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:48,641 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., pid=18, masterSystemTime=1682326728608 2023-04-24 08:58:48,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:48,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:48,643 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=415f076aa60cbee3cb48b662f1bba024, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:48,643 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326728643"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326728643"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326728643"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326728643"}]},"ts":"1682326728643"} 2023-04-24 08:58:48,647 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-24 08:58:48,647 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 in 192 msec 2023-04-24 08:58:48,650 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-24 08:58:48,650 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, ASSIGN in 356 msec 2023-04-24 08:58:48,650 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 08:58:48,651 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326728650"}]},"ts":"1682326728650"} 2023-04-24 08:58:48,652 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 08:58:48,683 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 08:58:48,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2890 sec 2023-04-24 08:58:49,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-24 08:58:49,506 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-24 08:58:49,507 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x10dd04e6 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:49,542 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4258ab17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:49,544 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:49,551 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:50792, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:49,553 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 08:58:49,557 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:39314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 08:58:49,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 08:58:49,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$12(2454): Client=jenkins//188.40.62.62 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 08:58:49,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-24 08:58:49,987 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 08:58:49,990 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 08:58:49,990 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 08:58:49,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-24 08:58:50,011 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, REOPEN/MOVE}] 2023-04-24 08:58:50,012 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, REOPEN/MOVE 2023-04-24 08:58:50,012 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=415f076aa60cbee3cb48b662f1bba024, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,013 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326730012"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326730012"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326730012"}]},"ts":"1682326730012"} 2023-04-24 08:58:50,014 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:58:50,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 415f076aa60cbee3cb48b662f1bba024, disabling compactions & flushes 2023-04-24 08:58:50,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. after waiting 0 ms 2023-04-24 08:58:50,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-24 08:58:50,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:50,172 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 415f076aa60cbee3cb48b662f1bba024 to self. 2023-04-24 08:58:50,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,174 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=415f076aa60cbee3cb48b662f1bba024, regionState=CLOSED 2023-04-24 08:58:50,174 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326730174"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326730174"}]},"ts":"1682326730174"} 2023-04-24 08:58:50,176 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-24 08:58:50,176 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 in 161 msec 2023-04-24 08:58:50,177 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=true 2023-04-24 08:58:50,327 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=415f076aa60cbee3cb48b662f1bba024, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,328 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326730327"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326730327"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326730327"}]},"ts":"1682326730327"} 2023-04-24 08:58:50,331 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:58:50,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 415f076aa60cbee3cb48b662f1bba024, NAME => 'TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:58:50,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:58:50,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,498 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,499 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A 2023-04-24 08:58:50,499 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A 2023-04-24 08:58:50,500 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:50,503 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 415f076aa60cbee3cb48b662f1bba024 columnFamilyName A 2023-04-24 08:58:50,505 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(310): Store=415f076aa60cbee3cb48b662f1bba024/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:50,506 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,507 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B 2023-04-24 08:58:50,507 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B 2023-04-24 08:58:50,508 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:50,508 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 415f076aa60cbee3cb48b662f1bba024 columnFamilyName B 2023-04-24 08:58:50,509 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(310): Store=415f076aa60cbee3cb48b662f1bba024/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:50,509 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,510 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C 2023-04-24 08:58:50,511 DEBUG [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C 2023-04-24 08:58:50,511 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:58:50,512 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 415f076aa60cbee3cb48b662f1bba024 columnFamilyName C 2023-04-24 08:58:50,513 INFO [StoreOpener-415f076aa60cbee3cb48b662f1bba024-1] regionserver.HStore(310): Store=415f076aa60cbee3cb48b662f1bba024/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:58:50,513 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,514 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 08:58:50,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 415f076aa60cbee3cb48b662f1bba024; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64764335, jitterRate=-0.03493620455265045}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 08:58:50,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:50,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., pid=23, masterSystemTime=1682326730484 2023-04-24 08:58:50,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,521 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,522 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=415f076aa60cbee3cb48b662f1bba024, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,522 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326730522"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326730522"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326730522"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326730522"}]},"ts":"1682326730522"} 2023-04-24 08:58:50,524 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-24 08:58:50,525 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 in 192 msec 2023-04-24 08:58:50,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-24 08:58:50,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, REOPEN/MOVE in 514 msec 2023-04-24 08:58:50,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-24 08:58:50,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 531 msec 2023-04-24 08:58:50,530 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 961 msec 2023-04-24 08:58:50,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-24 08:58:50,539 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x6dc16927 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,563 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65dbf2b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,564 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x029d68f7 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,575 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79385b7c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,575 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x6322d500 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,583 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a11c343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,583 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x4af56759 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,591 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e7aae9f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,591 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x5d07b2f5 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,600 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ca9d9a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,600 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x34b669b1 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,608 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46a0ca45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,609 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x6683e622 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,617 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61fff0a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,617 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x615d8f34 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,633 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1896d7cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,633 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x4890ce53 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:58:50,641 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45b9cc5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:58:50,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:50,650 DEBUG [hconnection-0x61e887e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,650 DEBUG [hconnection-0x5a6855f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,651 DEBUG [hconnection-0x129c53d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,653 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,653 DEBUG [hconnection-0x41f46182-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,659 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,659 DEBUG [hconnection-0x1839d49a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,659 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,661 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,664 DEBUG [hconnection-0x43401890-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:50,669 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:50,669 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:50,669 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:50,669 DEBUG [hconnection-0x71beea9c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,669 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:50,671 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,671 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45218, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,671 DEBUG [hconnection-0x37165897-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,671 DEBUG [hconnection-0x6401ad77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:58:50,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:50,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:50,674 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,678 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45246, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,679 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:45244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:58:50,682 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:50,682 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:50,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:50,683 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,683 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:50,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:50,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:50,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:50,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:50,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:50,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:50,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:50,692 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:50,692 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:50,692 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:50,692 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:50,695 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 08:58:50,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:50,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:50,699 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:58:50,699 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:50,699 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:58:50,699 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:50,699 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:58:50,699 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326730717, max timestamp=1682326730721Num uniques 859; 2023-04-24 08:58:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326730717, max timestamp=1682326730721Num uniques 859; 2023-04-24 08:58:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326730717, max timestamp=1682326730721Num uniques 859; 2023-04-24 08:58:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:58:50,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326790746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326790750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326790752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326790753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326790754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326790856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326790857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326790858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326790858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:50,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326790860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326791059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326791060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326791061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326791061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326791062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,171 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424322f6362daa3468383480763d1f4910f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424322f6362daa3468383480763d1f4910f 2023-04-24 08:58:51,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326791365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326791365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326791366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326791366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326791368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,582 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/cfb1a87d60c34ac0ab4c8b7b19ca40de 2023-04-24 08:58:51,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326791873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326791873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326791873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326791874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:51,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:51,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326791877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,002 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/067c91315e984683810b3b6041e50b7b 2023-04-24 08:58:52,118 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 08:58:52,423 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/6d266c46bc804b4789f6b185f41a12f9 2023-04-24 08:58:52,429 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/cfb1a87d60c34ac0ab4c8b7b19ca40de as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cfb1a87d60c34ac0ab4c8b7b19ca40de 2023-04-24 08:58:52,435 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cfb1a87d60c34ac0ab4c8b7b19ca40de, entries=150, sequenceid=14, filesize=25.1 K 2023-04-24 08:58:52,437 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/067c91315e984683810b3b6041e50b7b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/067c91315e984683810b3b6041e50b7b 2023-04-24 08:58:52,446 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/067c91315e984683810b3b6041e50b7b, entries=150, sequenceid=14, filesize=11.6 K 2023-04-24 08:58:52,447 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/6d266c46bc804b4789f6b185f41a12f9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/6d266c46bc804b4789f6b185f41a12f9 2023-04-24 08:58:52,456 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/6d266c46bc804b4789f6b185f41a12f9, entries=150, sequenceid=14, filesize=11.6 K 2023-04-24 08:58:52,458 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 415f076aa60cbee3cb48b662f1bba024 in 1762ms, sequenceid=14, compaction requested=false 2023-04-24 08:58:52,458 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:52,458 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:52,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:52,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:52,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:52,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:52,499 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:52,499 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:52,499 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:52,499 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:52,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:52,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c4bf4b8[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:52,501 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:52,501 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:52,507 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:52,507 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:52,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,507 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:52,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:52,507 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:52,515 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:52,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:52,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:52,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:52,519 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:52,519 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:52,519 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:52,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:52,520 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:52,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:52,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:52,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:52,529 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,530 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:52,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:52,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:52,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:52,541 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,541 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:52,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:52,541 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:52,542 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:52,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:52,642 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:52,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:52,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:52,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:52,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:52,647 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:52,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:52,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:52,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:52,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:52,657 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:52,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:52,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:52,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:52,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:52,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:52,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:52,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:52,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:52,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:52,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:52,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:52,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:52,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:52,672 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-24 08:58:52,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:58:52,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:52,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:58:52,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:52,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:58:52,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:52,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:52,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:52,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326792881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326792882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326792883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326792883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326792887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326792985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326792985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326792988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:52,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:52,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326792988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,089 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042419dfcd7fdc8f45f48c7f763457472398 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042419dfcd7fdc8f45f48c7f763457472398 2023-04-24 08:58:53,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:53,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326793187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:53,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326793190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:53,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326793193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:53,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326793193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:53,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326793490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,494 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/389f93ff27b148b79ab76f4318db7ed4 2023-04-24 08:58:53,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:53,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326793492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:53,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326793498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:53,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326793498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,509 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/96e5f79d37014b7996bf40698626b537 2023-04-24 08:58:53,520 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/73184a6c7ca444d297425d87eedfd935 2023-04-24 08:58:53,524 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/389f93ff27b148b79ab76f4318db7ed4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/389f93ff27b148b79ab76f4318db7ed4 2023-04-24 08:58:53,529 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/389f93ff27b148b79ab76f4318db7ed4, entries=150, sequenceid=51, filesize=25.1 K 2023-04-24 08:58:53,530 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/96e5f79d37014b7996bf40698626b537 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/96e5f79d37014b7996bf40698626b537 2023-04-24 08:58:53,535 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/96e5f79d37014b7996bf40698626b537, entries=150, sequenceid=51, filesize=11.6 K 2023-04-24 08:58:53,535 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/73184a6c7ca444d297425d87eedfd935 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/73184a6c7ca444d297425d87eedfd935 2023-04-24 08:58:53,545 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/73184a6c7ca444d297425d87eedfd935, entries=150, sequenceid=51, filesize=11.6 K 2023-04-24 08:58:53,546 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 415f076aa60cbee3cb48b662f1bba024 in 874ms, sequenceid=51, compaction requested=false 2023-04-24 08:58:53,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:53,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:53,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:53,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:53,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:53,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:53,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:53,558 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:53,558 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:53,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:53,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@692b3889[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:53,559 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:53,559 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:53,565 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:53,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:53,566 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,566 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:53,566 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:53,566 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:53,566 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:53,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:53,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:53,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:53,575 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:53,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:53,577 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:53,577 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:53,577 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:53,577 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:53,577 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:53,582 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,582 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,582 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:53,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,583 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:53,585 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:53,591 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:53,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:53,591 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:53,591 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:53,591 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:53,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:53,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:53,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:53,691 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:53,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:53,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:53,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:53,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:53,697 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:53,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:53,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:53,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:53,698 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,698 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:53,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:53,707 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:53,707 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:53,707 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:53,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:53,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:53,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:53,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:53,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:53,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:53,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:53,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:53,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:53,709 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:53,709 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:53,709 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:58:53,709 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:58:53,709 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:53,709 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:58:53,709 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:53,710 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:58:53,710 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:53,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:53,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:54,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326794023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326734017, max timestamp=1682326734026Num uniques 859; 2023-04-24 08:58:54,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:54,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326734017, max timestamp=1682326734026Num uniques 859; 2023-04-24 08:58:54,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:54,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326734017, max timestamp=1682326734026Num uniques 859; 2023-04-24 08:58:54,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:54,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326794054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326794054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326794057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,127 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304243f56accbe6c24cb2b6af10905c8c9788 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243f56accbe6c24cb2b6af10905c8c9788 2023-04-24 08:58:54,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326794130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,133 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/395f37034acd4abe9a9770832e0c23d2 2023-04-24 08:58:54,161 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/0f748a096c12424985e9f5714aaaaf7b 2023-04-24 08:58:54,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326794163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326794166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326794188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,219 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/711341c702d84b318d79e21d7fb9d520 2023-04-24 08:58:54,227 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/395f37034acd4abe9a9770832e0c23d2 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/395f37034acd4abe9a9770832e0c23d2 2023-04-24 08:58:54,238 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/395f37034acd4abe9a9770832e0c23d2, entries=150, sequenceid=61, filesize=25.1 K 2023-04-24 08:58:54,240 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/0f748a096c12424985e9f5714aaaaf7b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/0f748a096c12424985e9f5714aaaaf7b 2023-04-24 08:58:54,251 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/0f748a096c12424985e9f5714aaaaf7b, entries=150, sequenceid=61, filesize=11.6 K 2023-04-24 08:58:54,252 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/711341c702d84b318d79e21d7fb9d520 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/711341c702d84b318d79e21d7fb9d520 2023-04-24 08:58:54,258 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/711341c702d84b318d79e21d7fb9d520, entries=150, sequenceid=61, filesize=11.6 K 2023-04-24 08:58:54,263 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 415f076aa60cbee3cb48b662f1bba024 in 554ms, sequenceid=61, compaction requested=true 2023-04-24 08:58:54,263 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:54,263 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,263 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:54,263 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:54,263 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,263 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:54,263 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:54,274 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,274 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,274 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,274 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:54,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@788c4152[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:54,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:54,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,282 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:54,282 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,282 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:54,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:54,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:54,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:54,310 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:54,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:54,314 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:54,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:54,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:54,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:54,314 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:54,324 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,324 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,324 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,324 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:54,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:54,332 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:54,332 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:54,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:54,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,332 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:54,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-24 08:58:54,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:58:54,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:54,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:58:54,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:54,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:58:54,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:54,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326794365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326794374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326794388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326794391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,433 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:54,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:54,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:54,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:54,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:54,436 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:54,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:54,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:54,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:54,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,437 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:54,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:54,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:54,442 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,442 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:54,442 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:54,442 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:54,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:54,449 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,449 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,449 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,449 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:54,450 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:54,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bff09b[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:54,450 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:54,457 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:54,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:54,458 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:54,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,465 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,465 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,466 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:54,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:54,467 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:54,467 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:54,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:54,468 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:54,468 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:54,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,474 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,476 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326794476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,477 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,482 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:54,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:54,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:54,483 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:54,483 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:54,583 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:54,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:54,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:54,586 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:54,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:54,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:54,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:54,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,587 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,590 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,590 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,591 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,591 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:54,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:54,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:54,592 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,592 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:54,592 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:54,592 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:54,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:54,599 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,599 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,599 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,599 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:54,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75222ac0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:54,600 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:54,600 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,607 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:54,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:54,607 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,607 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:54,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:54,615 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,615 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,615 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,615 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:54,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:54,617 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:54,617 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:54,617 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:54,617 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:54,617 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:54,624 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,624 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,624 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,625 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,632 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,632 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,632 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,632 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:54,632 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,632 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:54,632 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,633 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,633 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:54,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,633 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326794677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326794677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326794694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326794695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,733 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:54,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:54,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:54,738 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:54,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:54,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:54,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:54,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,748 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,748 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,748 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,748 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:54,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:54,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:54,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:54,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:54,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:54,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:54,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:54,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:54,750 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,750 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:54,750 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:54,750 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:54,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:54,750 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,751 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:54,751 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:54,757 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,757 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,757 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:54,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@457dbb9d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:54,759 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:54,759 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,765 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,766 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,766 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:54,766 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:54,766 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:54,766 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:54,774 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,774 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,774 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,774 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,778 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042456cf97c68c83487180438d45a0d2f9c4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042456cf97c68c83487180438d45a0d2f9c4 2023-04-24 08:58:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:54,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:54,779 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:54,779 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:54,779 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:54,779 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:54,779 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:54,795 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/247b6c1f60e445769c9f49c3a04be36c 2023-04-24 08:58:54,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,813 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,814 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/171e4228b53c419eaa3e0b2938514825 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,824 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,824 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:54,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:54,825 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:54,825 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:54,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326794906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,908 DEBUG [Thread-778] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:54,925 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:54,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:54,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:54,928 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:54,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:54,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:54,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:54,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,932 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,932 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,932 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,932 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:54,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:54,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:54,934 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,934 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:54,934 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:54,934 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:54,934 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:54,940 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,940 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,941 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,941 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:54,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fe1f4b1[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:54,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:54,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,949 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,949 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,949 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:54,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:54,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:54,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:54,957 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:54,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,957 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,957 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:54,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:54,959 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:54,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:54,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:54,959 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:54,959 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:54,974 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,974 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,974 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:54,974 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:54,975 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:54,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:54,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:54,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:54,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:54,982 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:54,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:54,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:54,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:54,982 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:54,983 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,983 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:54,983 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:54,983 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:54,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:54,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:54,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326794983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,083 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:55,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:55,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:55,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:55,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:55,086 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:55,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:55,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:55,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:55,087 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,087 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,090 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:55,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:55,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:55,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:55,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:55,092 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:55,092 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:55,092 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:55,092 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:55,092 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:55,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,099 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:55,099 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,099 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:55,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:55,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2623fcfe[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:55,100 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:55,100 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,107 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:55,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:55,107 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,107 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:55,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:55,115 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,115 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:55,115 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:55,116 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:55,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:55,117 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:55,117 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:55,117 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:55,117 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:55,117 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:55,124 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,124 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:55,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,124 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:55,124 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,125 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,132 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:55,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:55,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:55,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:55,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:55,133 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:55,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:55,133 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:55,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,133 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:55,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:55,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326795183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:55,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326795199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:55,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326795201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,233 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:55,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:55,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:55,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:55,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:55,236 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:55,236 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:55,236 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:55,236 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:55,237 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,237 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,240 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,240 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:55,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:55,241 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,241 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:55,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:55,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:55,241 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:55,242 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:55,242 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:55,242 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:55,242 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:55,242 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:55,244 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/0c9ed9c66c324dfb97e94f68bee59ae0 2023-04-24 08:58:55,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/247b6c1f60e445769c9f49c3a04be36c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/247b6c1f60e445769c9f49c3a04be36c 2023-04-24 08:58:55,249 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,249 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:55,249 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,249 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4233a6d0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:55,250 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:55,250 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/247b6c1f60e445769c9f49c3a04be36c, entries=200, sequenceid=98, filesize=31.9 K 2023-04-24 08:58:55,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/171e4228b53c419eaa3e0b2938514825 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/171e4228b53c419eaa3e0b2938514825 2023-04-24 08:58:55,258 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,258 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:55,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:55,258 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:55,258 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:55,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/171e4228b53c419eaa3e0b2938514825, entries=150, sequenceid=98, filesize=11.6 K 2023-04-24 08:58:55,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/0c9ed9c66c324dfb97e94f68bee59ae0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/0c9ed9c66c324dfb97e94f68bee59ae0 2023-04-24 08:58:55,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:55,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:55,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:55,270 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:55,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:55,280 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:55,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:55,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:55,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:55,283 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:55,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/0c9ed9c66c324dfb97e94f68bee59ae0, entries=150, sequenceid=98, filesize=11.6 K 2023-04-24 08:58:55,285 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 415f076aa60cbee3cb48b662f1bba024 in 949ms, sequenceid=98, compaction requested=true 2023-04-24 08:58:55,285 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:55,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:55,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 08:58:55,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-24 08:58:55,285 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:55,287 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:55,287 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/A is initiating minor compaction (all files) 2023-04-24 08:58:55,287 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/A in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:55,287 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cfb1a87d60c34ac0ab4c8b7b19ca40de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/389f93ff27b148b79ab76f4318db7ed4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/395f37034acd4abe9a9770832e0c23d2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/247b6c1f60e445769c9f49c3a04be36c] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=107.3 K 2023-04-24 08:58:55,288 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting cfb1a87d60c34ac0ab4c8b7b19ca40de, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326730675 2023-04-24 08:58:55,288 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 389f93ff27b148b79ab76f4318db7ed4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682326730741 2023-04-24 08:58:55,289 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 395f37034acd4abe9a9770832e0c23d2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682326732881 2023-04-24 08:58:55,289 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 247b6c1f60e445769c9f49c3a04be36c, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326734046 2023-04-24 08:58:55,290 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:55,290 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,291 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:55,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,291 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,292 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,292 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:55,294 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:55,294 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/C is initiating minor compaction (all files) 2023-04-24 08:58:55,294 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/C in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:55,295 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/6d266c46bc804b4789f6b185f41a12f9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/73184a6c7ca444d297425d87eedfd935, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/711341c702d84b318d79e21d7fb9d520, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/0c9ed9c66c324dfb97e94f68bee59ae0] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=46.4 K 2023-04-24 08:58:55,295 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 6d266c46bc804b4789f6b185f41a12f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326730675 2023-04-24 08:58:55,296 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 73184a6c7ca444d297425d87eedfd935, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682326730741 2023-04-24 08:58:55,296 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 711341c702d84b318d79e21d7fb9d520, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682326732881 2023-04-24 08:58:55,297 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 0c9ed9c66c324dfb97e94f68bee59ae0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326734046 2023-04-24 08:58:55,313 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#A#compaction#111 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:55,315 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#C#compaction#112 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:55,320 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:55,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:55,320 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:55,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:55,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:55,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:55,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:55,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:55,325 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,326 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,326 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,326 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:55,326 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,326 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:55,326 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:55,368 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/8d6cb18eed46429fba33f94322107065 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8d6cb18eed46429fba33f94322107065 2023-04-24 08:58:55,374 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/C of 415f076aa60cbee3cb48b662f1bba024 into 8d6cb18eed46429fba33f94322107065(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:55,375 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:55,375 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/C, priority=12, startTime=1682326735285; duration=0sec 2023-04-24 08:58:55,375 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:55,375 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:55,377 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:55,377 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/B is initiating minor compaction (all files) 2023-04-24 08:58:55,378 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/B in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:55,378 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/067c91315e984683810b3b6041e50b7b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/96e5f79d37014b7996bf40698626b537, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/0f748a096c12424985e9f5714aaaaf7b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/171e4228b53c419eaa3e0b2938514825] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=46.4 K 2023-04-24 08:58:55,378 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 067c91315e984683810b3b6041e50b7b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326730675 2023-04-24 08:58:55,379 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 96e5f79d37014b7996bf40698626b537, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682326730741 2023-04-24 08:58:55,379 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 0f748a096c12424985e9f5714aaaaf7b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682326732881 2023-04-24 08:58:55,380 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 171e4228b53c419eaa3e0b2938514825, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326734046 2023-04-24 08:58:55,392 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#B#compaction#113 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:55,426 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:55,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:55,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:55,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:55,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:55,431 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:55,431 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:55,431 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:55,431 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:55,431 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,431 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:55,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:55,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:55,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:55,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:55,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:55,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:55,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:55,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:55,443 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:55,443 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:55,443 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:55,443 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:55,443 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 08:58:55,443 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:58:55,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:55,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:58:55,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:55,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:58:55,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:55,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:55,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:55,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326735524, max timestamp=1682326735539Num uniques 859; 2023-04-24 08:58:55,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:55,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326735524, max timestamp=1682326735539Num uniques 859; 2023-04-24 08:58:55,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:55,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326735524, max timestamp=1682326735539Num uniques 859; 2023-04-24 08:58:55,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:58:55,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:55,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326795596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:55,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326795699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:55,823 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/fe2ade4869b140f4a34e8f97ec5e251a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/fe2ade4869b140f4a34e8f97ec5e251a 2023-04-24 08:58:55,829 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/B of 415f076aa60cbee3cb48b662f1bba024 into fe2ade4869b140f4a34e8f97ec5e251a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:55,829 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:55,829 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/B, priority=12, startTime=1682326735285; duration=0sec 2023-04-24 08:58:55,830 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:55,872 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424fbcff9ed067d43018129d9fa6dbb34fb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fbcff9ed067d43018129d9fa6dbb34fb 2023-04-24 08:58:55,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:55,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326795902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:56,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326796200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326796205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:56,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326796207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326796209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,279 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/d25db1a8613c4d178636a2f14454b261 2023-04-24 08:58:56,556 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/cb9b58fb65ba4fdfb4cb2a038a6f90be as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cb9b58fb65ba4fdfb4cb2a038a6f90be 2023-04-24 08:58:56,560 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/A of 415f076aa60cbee3cb48b662f1bba024 into cb9b58fb65ba4fdfb4cb2a038a6f90be(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-24 08:58:56,560 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:56,560 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/A, priority=12, startTime=1682326735285; duration=1sec 2023-04-24 08:58:56,560 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:56,695 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/9b0717842662431db1e64e49cd3a3cca 2023-04-24 08:58:56,710 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/15af92ad29ca44f39052b6d5922cf0d1 2023-04-24 08:58:56,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:56,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326796714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,717 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/d25db1a8613c4d178636a2f14454b261 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d25db1a8613c4d178636a2f14454b261 2023-04-24 08:58:56,722 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d25db1a8613c4d178636a2f14454b261, entries=100, sequenceid=108, filesize=18.4 K 2023-04-24 08:58:56,724 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/9b0717842662431db1e64e49cd3a3cca as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/9b0717842662431db1e64e49cd3a3cca 2023-04-24 08:58:56,730 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/9b0717842662431db1e64e49cd3a3cca, entries=100, sequenceid=108, filesize=9.3 K 2023-04-24 08:58:56,731 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/15af92ad29ca44f39052b6d5922cf0d1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/15af92ad29ca44f39052b6d5922cf0d1 2023-04-24 08:58:56,740 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/15af92ad29ca44f39052b6d5922cf0d1, entries=100, sequenceid=108, filesize=9.3 K 2023-04-24 08:58:56,740 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 415f076aa60cbee3cb48b662f1bba024 in 1297ms, sequenceid=108, compaction requested=false 2023-04-24 08:58:56,741 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:56,741 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:56,741 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:56,741 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:56,741 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,741 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:56,741 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:56,774 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:56,774 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:56,775 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:56,775 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:56,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:56,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aba5c58[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:56,776 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:56,776 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:56,782 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:56,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:56,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:56,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:56,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:56,783 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,783 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:56,790 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,790 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:56,790 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:56,791 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:56,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:56,793 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:56,793 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:56,793 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:56,793 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:56,793 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:56,799 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:56,799 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,799 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:56,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:56,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:56,801 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:56,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,807 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,807 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:56,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:56,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:56,807 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:56,807 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:56,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:56,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:56,807 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:56,808 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:56,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:56,808 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:56,808 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:56,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:56,908 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:56,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:56,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:56,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:56,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:56,911 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:56,911 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:56,911 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:56,911 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:56,911 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,911 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:56,915 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,915 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:56,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:56,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:56,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:56,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:56,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:56,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:56,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:56,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:56,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:56,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:56,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:56,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:56,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:56,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:56,918 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-24 08:58:56,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:58:56,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:56,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:58:56,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:56,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:58:56,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:57,329 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424aefccf06ec8d424ba42e693d716cc048 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424aefccf06ec8d424ba42e693d716cc048 2023-04-24 08:58:57,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:57,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:57,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326797738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:57,745 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/71091321b3624cd99fec70ad8478a951 2023-04-24 08:58:57,766 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/eb57e38a77cc45adb23cfda15635309d 2023-04-24 08:58:57,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:57,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326797843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326798046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,179 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/94da91d9868a4eaea7deaf4760d4f7a8 2023-04-24 08:58:58,186 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/71091321b3624cd99fec70ad8478a951 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/71091321b3624cd99fec70ad8478a951 2023-04-24 08:58:58,191 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/71091321b3624cd99fec70ad8478a951, entries=150, sequenceid=147, filesize=25.3 K 2023-04-24 08:58:58,193 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/eb57e38a77cc45adb23cfda15635309d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/eb57e38a77cc45adb23cfda15635309d 2023-04-24 08:58:58,198 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/eb57e38a77cc45adb23cfda15635309d, entries=150, sequenceid=147, filesize=11.8 K 2023-04-24 08:58:58,199 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/94da91d9868a4eaea7deaf4760d4f7a8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/94da91d9868a4eaea7deaf4760d4f7a8 2023-04-24 08:58:58,206 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/94da91d9868a4eaea7deaf4760d4f7a8, entries=150, sequenceid=147, filesize=11.8 K 2023-04-24 08:58:58,207 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 415f076aa60cbee3cb48b662f1bba024 in 1289ms, sequenceid=147, compaction requested=true 2023-04-24 08:58:58,207 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:58,207 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:58,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:58,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:58,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:58,215 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,215 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,216 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,216 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:58,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a2e69c2[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:58,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:58,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:58,221 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:58:58,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:58:58,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:58,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:58:58,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:58,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:58:58,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:58:58,224 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,224 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:58,224 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:58,224 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:58,232 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:58,232 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,232 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:58,232 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:58,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:58,233 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:58,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:58,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:58,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:58,233 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:58,240 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,241 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,241 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,245 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,247 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326738244, max timestamp=1682326738251Num uniques 859; 2023-04-24 08:58:58,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 08:58:58,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326738244, max timestamp=1682326738251Num uniques 859; 2023-04-24 08:58:58,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 08:58:58,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326738244, max timestamp=1682326738251Num uniques 859; 2023-04-24 08:58:58,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,257 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,257 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:58,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:58,258 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:58,258 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:58,264 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304242b963a76fc004f72a250264423d0a3f9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242b963a76fc004f72a250264423d0a3f9 2023-04-24 08:58:58,269 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/bad0e709fb5e4db790c15cb5eb16d754 2023-04-24 08:58:58,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326798271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326798276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326798277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326798352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,358 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:58,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:58,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:58,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:58,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:58,365 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:58,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:58,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:58,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:58,366 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,366 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,374 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,374 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,374 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:58,376 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:58,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:58,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:58,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:58,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:58,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:58,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:58,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:58,378 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,378 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:58,378 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:58,379 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,379 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:58,379 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:58,379 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,379 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:58,379 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:58,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326798378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,382 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:58,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326798381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:58,384 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:58,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3b675[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:58,384 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326798381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,390 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,390 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,391 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,391 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:58,391 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:58,391 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:58,399 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:58,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,399 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,399 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:58,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:58,401 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:58,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:58,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:58,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:58,401 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:58,407 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,407 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,407 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,410 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,415 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:58,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:58,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,416 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:58,416 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:58,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,416 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,516 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:58,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:58,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:58,519 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:58,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:58,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:58,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:58,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,543 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,543 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:58,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:58,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:58,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:58,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:58,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:58,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:58,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:58,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:58,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0985f9[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:58,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:58,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,565 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,565 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,566 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:58,566 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:58,566 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:58,566 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:58,574 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:58,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:58,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:58,576 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:58,576 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:58,576 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:58,576 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:58,576 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:58,582 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,582 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,582 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,582 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326798582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,587 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326798587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326798589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,599 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,599 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,599 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:58,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:58,599 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,599 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,599 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,599 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:58,599 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,600 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:58,600 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,600 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,600 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/a0267282fcfc43519a7965d9b0a7cf7d 2023-04-24 08:58:58,700 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:58,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:58,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:58,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:58,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:58,703 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:58,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:58,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:58,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:58,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,707 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,707 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,707 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:58,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:58,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:58,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:58,708 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:58,708 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:58,708 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:58,708 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:58,708 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:58,708 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,708 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:58,708 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:58,708 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:58,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:58,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:58,709 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:58,715 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,715 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,716 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,716 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:58,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e94abc[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:58,717 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:58,717 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,724 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,724 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:58,724 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:58,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:58,724 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:58,732 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:58,732 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,732 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,732 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:58,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:58,734 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:58,734 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:58,734 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:58,734 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:58,734 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:58,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,743 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:58,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:58,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:58,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,749 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,749 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:58,749 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,849 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:58,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:58,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:58,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:58,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:58,852 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:58,852 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:58,852 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:58,852 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:58,853 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,853 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326798854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:58,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:58,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:58,858 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,858 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:58,858 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:58,858 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:58,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:58,865 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,866 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,866 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:58,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d404f9a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:58,867 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:58,867 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,874 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,874 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:58,874 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,874 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:58,874 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:58,874 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:58,882 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:58,882 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,882 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,882 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:58,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:58,884 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:58,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:58,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:58,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:58,884 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:58,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326798886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,890 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,890 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,890 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:58,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:58,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:58,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326798890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326798893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,899 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:58,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:58,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:58,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:58,899 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,899 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:58,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:58,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:58,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:58,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326798927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:58,930 DEBUG [Thread-778] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:58:58,999 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:59,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:59,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:59,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:59,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:59,002 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:59,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:59,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:59,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:59,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:59,007 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:59,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:59,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:59,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:59,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:59,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:59,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:59,008 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:59,008 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:59,009 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:59,009 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:59,009 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:59,009 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:59,009 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:59,009 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:59,009 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:59,009 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:58:59,009 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:58:59,009 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,009 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:58:59,009 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:58:59,035 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,035 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:59,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:59,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:58:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:59,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:59,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:58:59,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@333e3459[Count = 0] remaining members to acquire global barrier 2023-04-24 08:58:59,037 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:58:59,037 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:59,049 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:59,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:59,049 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:58:59,049 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:58:59,049 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,049 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:58:59,049 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:58:59,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:58:59,057 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:59,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:58:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:58:59,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:58:59,059 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:58:59,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:58:59,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:58:59,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:58:59,059 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:58:59,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:58:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:58:59,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:58:59,066 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:58:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:59,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:58:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:58:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:59,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:59,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:58:59,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:58:59,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:58:59,074 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:58:59,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:58:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:58:59,074 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:58:59,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:58:59,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:59,074 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:58:59,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:59,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:59,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:58:59,075 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,089 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/be8c0419fb0f40bea0c9981effe71472 2023-04-24 08:58:59,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/bad0e709fb5e4db790c15cb5eb16d754 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/bad0e709fb5e4db790c15cb5eb16d754 2023-04-24 08:58:59,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/bad0e709fb5e4db790c15cb5eb16d754, entries=200, sequenceid=157, filesize=32.1 K 2023-04-24 08:58:59,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/a0267282fcfc43519a7965d9b0a7cf7d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/a0267282fcfc43519a7965d9b0a7cf7d 2023-04-24 08:58:59,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/a0267282fcfc43519a7965d9b0a7cf7d, entries=150, sequenceid=157, filesize=11.8 K 2023-04-24 08:58:59,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/be8c0419fb0f40bea0c9981effe71472 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/be8c0419fb0f40bea0c9981effe71472 2023-04-24 08:58:59,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/be8c0419fb0f40bea0c9981effe71472, entries=150, sequenceid=157, filesize=11.8 K 2023-04-24 08:58:59,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 415f076aa60cbee3cb48b662f1bba024 in 886ms, sequenceid=157, compaction requested=true 2023-04-24 08:58:59,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:59,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:59,108 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:59,108 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:59,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:59,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:59,109 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103464 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:59,109 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45638 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:59,109 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/A is initiating minor compaction (all files) 2023-04-24 08:58:59,109 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/B is initiating minor compaction (all files) 2023-04-24 08:58:59,109 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/A in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:59,109 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/B in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:59,109 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cb9b58fb65ba4fdfb4cb2a038a6f90be, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d25db1a8613c4d178636a2f14454b261, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/71091321b3624cd99fec70ad8478a951, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/bad0e709fb5e4db790c15cb5eb16d754] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=101.0 K 2023-04-24 08:58:59,109 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/fe2ade4869b140f4a34e8f97ec5e251a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/9b0717842662431db1e64e49cd3a3cca, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/eb57e38a77cc45adb23cfda15635309d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/a0267282fcfc43519a7965d9b0a7cf7d] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=44.6 K 2023-04-24 08:58:59,110 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting cb9b58fb65ba4fdfb4cb2a038a6f90be, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326734046 2023-04-24 08:58:59,110 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting fe2ade4869b140f4a34e8f97ec5e251a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326734046 2023-04-24 08:58:59,110 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting d25db1a8613c4d178636a2f14454b261, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682326734346 2023-04-24 08:58:59,110 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 9b0717842662431db1e64e49cd3a3cca, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682326734346 2023-04-24 08:58:59,111 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting eb57e38a77cc45adb23cfda15635309d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682326735579 2023-04-24 08:58:59,111 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 71091321b3624cd99fec70ad8478a951, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682326735579 2023-04-24 08:58:59,111 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a0267282fcfc43519a7965d9b0a7cf7d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682326737728 2023-04-24 08:58:59,111 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting bad0e709fb5e4db790c15cb5eb16d754, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682326737728 2023-04-24 08:58:59,118 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#B#compaction#123 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:59,122 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#A#compaction#124 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:59,174 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:58:59,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:58:59,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:58:59,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:58:59,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:58:59,177 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:58:59,177 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:59,177 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:58:59,177 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:58:59,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:59,190 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:58:59,190 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:58:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:58:59,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:58:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:59,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:58:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:58:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:58:59,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:58:59,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:58:59,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:58:59,192 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:58:59,192 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:58:59,192 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:58:59,192 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:58:59,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:59,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:58:59,192 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:58:59,192 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-24 08:58:59,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:58:59,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:59,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:58:59,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:59,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:58:59,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:58:59,209 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424b1129015650f49cd8d01219cbb74abea to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b1129015650f49cd8d01219cbb74abea 2023-04-24 08:58:59,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:58:59,392 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:58:59,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326799402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326799405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326799405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326799506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326799508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326799509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,529 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/c0291c94954043ec88af748edb0c5789 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c0291c94954043ec88af748edb0c5789 2023-04-24 08:58:59,535 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/B of 415f076aa60cbee3cb48b662f1bba024 into c0291c94954043ec88af748edb0c5789(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:59,535 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:59,535 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/B, priority=12, startTime=1682326739108; duration=0sec 2023-04-24 08:58:59,535 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:58:59,535 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:58:59,537 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45638 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:58:59,537 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/C is initiating minor compaction (all files) 2023-04-24 08:58:59,537 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/C in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:58:59,537 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8d6cb18eed46429fba33f94322107065, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/15af92ad29ca44f39052b6d5922cf0d1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/94da91d9868a4eaea7deaf4760d4f7a8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/be8c0419fb0f40bea0c9981effe71472] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=44.6 K 2023-04-24 08:58:59,538 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 8d6cb18eed46429fba33f94322107065, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326734046 2023-04-24 08:58:59,538 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 15af92ad29ca44f39052b6d5922cf0d1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682326734346 2023-04-24 08:58:59,539 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 94da91d9868a4eaea7deaf4760d4f7a8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1682326735579 2023-04-24 08:58:59,539 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting be8c0419fb0f40bea0c9981effe71472, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682326737728 2023-04-24 08:58:59,543 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/e183dd4063c946d9870109d65bd14677 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/e183dd4063c946d9870109d65bd14677 2023-04-24 08:58:59,548 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/A of 415f076aa60cbee3cb48b662f1bba024 into e183dd4063c946d9870109d65bd14677(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:59,548 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:59,548 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/A, priority=12, startTime=1682326739108; duration=0sec 2023-04-24 08:58:59,548 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:58:59,553 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#C#compaction#126 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:58:59,615 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=192, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/dc89408762aa43179b982e15b65c9613 2023-04-24 08:58:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326799710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326799710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326799710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:58:59,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326799860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:58:59,962 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/a0f3c024073d4944be26b5f4cf95978a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a0f3c024073d4944be26b5f4cf95978a 2023-04-24 08:58:59,967 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/C of 415f076aa60cbee3cb48b662f1bba024 into a0f3c024073d4944be26b5f4cf95978a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:58:59,967 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:58:59,967 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/C, priority=12, startTime=1682326739108; duration=0sec 2023-04-24 08:58:59,967 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:00,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326800012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326800013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326800013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,039 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/abe1cc5031e940a5bfef341adf56d9f3 2023-04-24 08:59:00,050 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/431baf35819c4de1b4721f1bbd725af2 2023-04-24 08:59:00,055 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/dc89408762aa43179b982e15b65c9613 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/dc89408762aa43179b982e15b65c9613 2023-04-24 08:59:00,060 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/dc89408762aa43179b982e15b65c9613, entries=150, sequenceid=192, filesize=25.3 K 2023-04-24 08:59:00,061 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/abe1cc5031e940a5bfef341adf56d9f3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/abe1cc5031e940a5bfef341adf56d9f3 2023-04-24 08:59:00,065 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/abe1cc5031e940a5bfef341adf56d9f3, entries=150, sequenceid=192, filesize=11.8 K 2023-04-24 08:59:00,066 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/431baf35819c4de1b4721f1bbd725af2 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/431baf35819c4de1b4721f1bbd725af2 2023-04-24 08:59:00,071 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/431baf35819c4de1b4721f1bbd725af2, entries=150, sequenceid=192, filesize=11.8 K 2023-04-24 08:59:00,072 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=53.67 KB/54960 for 415f076aa60cbee3cb48b662f1bba024 in 880ms, sequenceid=192, compaction requested=false 2023-04-24 08:59:00,072 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:00,072 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:00,107 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:00,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:00,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:00,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:00,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2308598a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:00,108 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:00,108 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:00,115 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:00,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:00,115 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:00,115 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:00,115 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,116 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:00,115 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:00,123 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:00,124 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:00,123 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,124 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:00,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:00,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:00,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:00,126 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:00,126 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:00,126 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:00,126 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:00,126 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:00,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:00,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,132 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:00,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:00,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:00,133 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,134 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,140 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,140 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:00,140 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:00,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:00,140 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:00,140 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:00,140 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:00,140 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,141 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:00,141 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:00,141 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:00,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:00,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:00,141 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:00,141 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:00,241 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:00,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:00,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:00,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:00,244 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:00,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:00,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:00,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:00,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:00,249 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,249 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:00,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:00,249 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,249 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:00,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:00,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:00,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:00,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:00,250 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:00,250 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:00,250 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:00,250 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:00,250 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:00,250 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:00,250 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:00,250 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 08:59:00,250 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:00,250 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:00,251 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:00,251 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:00,251 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:00,251 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:00,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:00,518 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:00,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326740533, max timestamp=1682326740540Num uniques 859; 2023-04-24 08:59:00,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:00,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326740533, max timestamp=1682326740540Num uniques 859; 2023-04-24 08:59:00,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:00,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326740533, max timestamp=1682326740540Num uniques 859; 2023-04-24 08:59:00,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:00,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326800564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326800566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326800566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,664 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424bd93e92323c44ada849490e3eb300d87 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bd93e92323c44ada849490e3eb300d87 2023-04-24 08:59:00,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326800668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326800670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326800670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326800872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326800874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:00,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:00,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326800874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,070 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/d38d093fa7d04666bc52a3023645f622 2023-04-24 08:59:01,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:01,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326801176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:01,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326801178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:01,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326801179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,483 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/be71fff4444a4509a8db7656462c024b 2023-04-24 08:59:01,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:01,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326801679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:01,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326801682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:01,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326801684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:01,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326801868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,871 DEBUG [Thread-782] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:59:01,894 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/3a1b7c9c588f437c8ab65d02591d3817 2023-04-24 08:59:01,897 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/d38d093fa7d04666bc52a3023645f622 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d38d093fa7d04666bc52a3023645f622 2023-04-24 08:59:01,901 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d38d093fa7d04666bc52a3023645f622, entries=150, sequenceid=206, filesize=25.3 K 2023-04-24 08:59:01,902 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/be71fff4444a4509a8db7656462c024b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/be71fff4444a4509a8db7656462c024b 2023-04-24 08:59:01,906 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/be71fff4444a4509a8db7656462c024b, entries=150, sequenceid=206, filesize=11.8 K 2023-04-24 08:59:01,907 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/3a1b7c9c588f437c8ab65d02591d3817 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3a1b7c9c588f437c8ab65d02591d3817 2023-04-24 08:59:01,912 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3a1b7c9c588f437c8ab65d02591d3817, entries=150, sequenceid=206, filesize=11.8 K 2023-04-24 08:59:01,913 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 415f076aa60cbee3cb48b662f1bba024 in 1663ms, sequenceid=206, compaction requested=true 2023-04-24 08:59:01,913 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:01,913 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:01,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:01,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:01,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,916 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:01,916 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:01,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:01,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:01,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:01,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:01,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d1a4bfd[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:01,984 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:01,984 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:01,990 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:01,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:01,990 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:01,990 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:01,990 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:01,990 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,991 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:01,998 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:01,998 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,998 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:01,999 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:02,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:02,000 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:02,000 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:02,001 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:02,001 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:02,001 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:02,007 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,007 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:02,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,007 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:02,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:02,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:02,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:02,015 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,015 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:02,015 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:02,015 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:02,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:02,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:02,015 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:02,015 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,015 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:02,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:02,016 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:02,016 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:02,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:02,015 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:02,015 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:02,016 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:02,116 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:02,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:02,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:02,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:02,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:02,119 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:02,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:02,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:02,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:02,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:02,123 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:02,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:02,124 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:02,124 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,124 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:02,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:02,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:02,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:02,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:02,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:02,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:02,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:02,127 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:02,127 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:02,127 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:02,127 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:02,127 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-24 08:59:02,127 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:02,127 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:02,127 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:02,128 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:02,128 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:02,128 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:02,139 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424746bcd6cc9ee4c4fbab87c98adf4c6a1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424746bcd6cc9ee4c4fbab87c98adf4c6a1 2023-04-24 08:59:02,545 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/eb15d316bf314034b2f6d692f63e29c8 2023-04-24 08:59:02,555 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/062641df850843cea6218ff5f33268c5 2023-04-24 08:59:02,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:02,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:02,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:02,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326802699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:02,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326802701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:02,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326802701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:02,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326802802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:02,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326802804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:02,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326802804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,965 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/eecaeb0e96ec4d8aaae784853794827c 2023-04-24 08:59:02,969 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/eb15d316bf314034b2f6d692f63e29c8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/eb15d316bf314034b2f6d692f63e29c8 2023-04-24 08:59:02,973 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/eb15d316bf314034b2f6d692f63e29c8, entries=150, sequenceid=243, filesize=25.3 K 2023-04-24 08:59:02,973 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/062641df850843cea6218ff5f33268c5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/062641df850843cea6218ff5f33268c5 2023-04-24 08:59:02,977 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/062641df850843cea6218ff5f33268c5, entries=150, sequenceid=243, filesize=11.8 K 2023-04-24 08:59:02,978 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/eecaeb0e96ec4d8aaae784853794827c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/eecaeb0e96ec4d8aaae784853794827c 2023-04-24 08:59:02,982 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/eecaeb0e96ec4d8aaae784853794827c, entries=150, sequenceid=243, filesize=11.8 K 2023-04-24 08:59:02,982 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 415f076aa60cbee3cb48b662f1bba024 in 855ms, sequenceid=243, compaction requested=true 2023-04-24 08:59:02,983 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:02,983 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:02,983 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:02,983 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:02,983 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,983 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:02,983 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:02,998 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,998 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:02,999 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:02,999 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:03,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2209f95b[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:03,000 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:03,000 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:03,005 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 08:59:03,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:03,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:03,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:03,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:03,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:03,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:03,007 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:03,008 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:03,008 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:03,008 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:03,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326803025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326803026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326803026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:03,036 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,037 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:03,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:03,039 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:03,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:03,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:03,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:03,039 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:03,090 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,090 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,090 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,092 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,098 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,098 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:03,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:03,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,099 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:03,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,099 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:03,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,099 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,099 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326803129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326803131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326803131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,199 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:03,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:03,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:03,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:03,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:03,202 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:03,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:03,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:03,203 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,203 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,207 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,207 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,207 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,207 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:03,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:03,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:03,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:03,208 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,208 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:03,208 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:03,208 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:03,208 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:03,215 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,215 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,215 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,215 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:03,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:03,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60578dbf[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:03,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,223 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,224 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:03,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:03,224 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:03,224 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:03,232 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:03,232 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,232 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,232 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:03,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:03,234 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:03,234 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:03,234 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:03,234 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:03,234 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:03,240 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,240 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,240 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,240 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,242 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,248 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,248 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,248 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,248 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:03,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:03,249 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:03,249 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,249 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:03,249 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,249 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:03,249 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,249 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,249 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326803332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326803333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326803336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,349 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:03,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:03,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:03,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:03,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:03,352 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:03,352 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,352 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:03,352 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:03,352 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,352 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,357 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,357 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,357 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:03,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:03,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:03,358 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:03,358 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,358 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:03,358 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:03,358 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:03,358 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,358 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:03,359 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:03,359 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,359 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:03,359 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:03,359 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,359 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:03,359 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:03,365 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,365 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,365 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,365 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:03,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:03,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f24c36[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:03,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,373 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,374 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,374 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:03,374 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:03,374 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:03,374 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:03,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:03,382 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:03,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:03,384 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:03,384 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:03,384 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:03,384 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:03,384 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:03,390 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,390 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,390 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,391 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,392 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,398 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,398 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,398 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,399 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,398 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:03,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:03,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,399 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:03,399 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:03,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,416 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424b2f3459ca2304d73910f28b66e437ec2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b2f3459ca2304d73910f28b66e437ec2 2023-04-24 08:59:03,499 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:03,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:03,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:03,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:03,502 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:03,502 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,502 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:03,502 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:03,503 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,503 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,507 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:03,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:03,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:03,508 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:03,508 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,508 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:03,508 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:03,508 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:03,508 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,508 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:03,509 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:03,509 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,509 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:03,509 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:03,509 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,509 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:03,509 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:03,515 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:03,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ee8aa56[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:03,517 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:03,517 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,523 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:03,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:03,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:03,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:03,532 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,532 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,532 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,532 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:03,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:03,534 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:03,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:03,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:03,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:03,534 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:03,540 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,540 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,540 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,541 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,542 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:03,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:03,549 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,549 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:03,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,549 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:03,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326803635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326803638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:03,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326803641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,649 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:03,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:03,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:03,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:03,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:03,652 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:03,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:03,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:03,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,653 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,657 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:03,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:03,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:03,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:03,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:03,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:03,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:03,658 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,658 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:03,658 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:03,659 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:03,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:03,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:03,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:03,665 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,665 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,666 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,666 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:03,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c643f1[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:03,666 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:03,667 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,673 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:03,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:03,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:03,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:03,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:03,682 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:03,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:03,684 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:03,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:03,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:03,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:03,684 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:03,690 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,690 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,690 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,690 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,707 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:03,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:03,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,707 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:03,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,707 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:03,707 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,808 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:03,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:03,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:03,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:03,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:03,811 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:03,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:03,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:03,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,815 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:03,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:03,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:03,817 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,817 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:03,817 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:03,817 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:03,817 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:03,821 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=255, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/2981b8096c4043359f5a7e64b1ab3b94 2023-04-24 08:59:03,823 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,824 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,824 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:03,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f201a43[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:03,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:03,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,832 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,832 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:03,832 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:03,832 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:03,833 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,833 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:03,840 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:03,840 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,840 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,840 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:03,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:03,842 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:03,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:03,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:03,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:03,842 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:03,848 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,848 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,848 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,849 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:03,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,857 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,857 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:03,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,958 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:03,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:03,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:03,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:03,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:03,961 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:03,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:03,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:03,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,965 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,965 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:03,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:03,965 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,965 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:03,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:03,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:03,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:03,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:03,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:03,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:03,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:03,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:03,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:03,966 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,966 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:03,966 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:03,966 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:03,967 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:03,967 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:03,967 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,967 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:03,967 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:03,973 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,973 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:03,975 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:03,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c20786d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:03,975 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:03,982 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,982 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:03,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:03,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:03,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:03,990 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:03,990 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,990 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,990 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:03,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:03,992 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:03,992 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:03,992 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:03,992 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:03,992 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:03,998 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,998 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,998 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:03,999 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:03,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,000 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,024 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,024 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,024 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,024 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:04,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:04,024 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,024 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,024 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:04,025 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:04,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,025 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:04,025 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,125 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:04,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:04,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:04,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:04,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:04,128 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:04,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:04,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:04,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,140 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,140 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,141 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,141 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:04,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326804141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:04,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45182 deadline: 1682326804141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:04,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:04,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:04,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:04,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:04,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:04,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:04,142 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,142 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:04,142 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:04,142 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,142 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:04,143 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:04,143 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,143 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:04,143 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:04,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:04,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45208 deadline: 1682326804146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,148 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,148 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,149 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,149 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:04,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6781c7a5[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:04,150 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:04,150 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,157 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,157 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:04,157 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:04,157 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:04,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:04,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:04,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,165 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:04,167 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:04,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:04,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:04,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:04,168 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:04,173 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,173 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,174 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,174 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,182 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:04,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,182 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,182 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:04,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,233 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/84df5cae824e47cd85ad2ce1845f935d 2023-04-24 08:59:04,283 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:04,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:04,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:04,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:04,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:04,285 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:04,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:04,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:04,286 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,286 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,290 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,290 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,290 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,290 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:04,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:04,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:04,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:04,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:04,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:04,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:04,291 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:04,292 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,292 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:04,292 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:04,292 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:04,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:04,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:04,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:04,298 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,298 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,299 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,299 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:04,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d9c96f[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:04,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:04,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,307 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:04,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:04,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:04,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:04,315 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:04,315 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,315 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,315 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:04,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:04,317 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:04,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:04,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:04,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:04,317 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:04,323 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,323 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,323 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,324 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,332 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:04,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:04,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:04,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,332 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:04,333 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:04,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,333 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,433 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:04,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:04,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:04,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:04,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:04,435 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:04,435 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,435 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:04,435 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:04,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:04,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:04,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:04,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:04,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:04,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:04,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:04,442 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,442 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:04,442 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:04,442 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:04,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:04,442 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,443 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:04,443 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:04,475 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,475 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,476 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,476 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:04,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:04,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@94435f5[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:04,480 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:04,480 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,490 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:04,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:04,491 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:04,491 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:04,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:04,498 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,499 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:04,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:04,500 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:04,501 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:04,501 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:04,501 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:04,501 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:04,507 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,507 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,507 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,508 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,534 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,534 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:04,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:04,534 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,534 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,534 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:04,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:04,534 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,534 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:04,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,534 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,534 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,534 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,634 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:04,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:04,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:04,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:04,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:04,637 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:04,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:04,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:04,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/aa2c1c81a09b48acb830684d990bee23 2023-04-24 08:59:04,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/2981b8096c4043359f5a7e64b1ab3b94 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/2981b8096c4043359f5a7e64b1ab3b94 2023-04-24 08:59:04,648 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,649 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,649 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,649 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:04,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:04,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:04,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:04,650 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:04,650 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,650 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:04,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/2981b8096c4043359f5a7e64b1ab3b94, entries=200, sequenceid=255, filesize=32.1 K 2023-04-24 08:59:04,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:04,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:04,651 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,651 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:04,651 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:04,651 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:04,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:04,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:04,651 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:04,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/84df5cae824e47cd85ad2ce1845f935d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/84df5cae824e47cd85ad2ce1845f935d 2023-04-24 08:59:04,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/84df5cae824e47cd85ad2ce1845f935d, entries=150, sequenceid=255, filesize=11.8 K 2023-04-24 08:59:04,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/aa2c1c81a09b48acb830684d990bee23 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/aa2c1c81a09b48acb830684d990bee23 2023-04-24 08:59:04,657 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,657 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,657 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,657 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:04,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7658c7a0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:04,659 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:04,659 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/aa2c1c81a09b48acb830684d990bee23, entries=150, sequenceid=255, filesize=11.8 K 2023-04-24 08:59:04,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 415f076aa60cbee3cb48b662f1bba024 in 1658ms, sequenceid=255, compaction requested=true 2023-04-24 08:59:04,663 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:04,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:04,663 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:59:04,663 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:59:04,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:04,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:04,665 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,665 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:59:04,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,665 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/B is initiating minor compaction (all files) 2023-04-24 08:59:04,665 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,665 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,665 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/B in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,666 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:04,666 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:04,666 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:04,666 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c0291c94954043ec88af748edb0c5789, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/abe1cc5031e940a5bfef341adf56d9f3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/be71fff4444a4509a8db7656462c024b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/062641df850843cea6218ff5f33268c5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/84df5cae824e47cd85ad2ce1845f935d] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=59.0 K 2023-04-24 08:59:04,666 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136744 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:59:04,667 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/A is initiating minor compaction (all files) 2023-04-24 08:59:04,667 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/A in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,667 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c0291c94954043ec88af748edb0c5789, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682326737728 2023-04-24 08:59:04,667 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/e183dd4063c946d9870109d65bd14677, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/dc89408762aa43179b982e15b65c9613, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d38d093fa7d04666bc52a3023645f622, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/eb15d316bf314034b2f6d692f63e29c8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/2981b8096c4043359f5a7e64b1ab3b94] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=133.5 K 2023-04-24 08:59:04,667 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting abe1cc5031e940a5bfef341adf56d9f3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682326738270 2023-04-24 08:59:04,667 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e183dd4063c946d9870109d65bd14677, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682326737728 2023-04-24 08:59:04,667 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting be71fff4444a4509a8db7656462c024b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682326739400 2023-04-24 08:59:04,668 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting dc89408762aa43179b982e15b65c9613, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682326738270 2023-04-24 08:59:04,668 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 062641df850843cea6218ff5f33268c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682326740558 2023-04-24 08:59:04,668 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting d38d093fa7d04666bc52a3023645f622, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682326739400 2023-04-24 08:59:04,668 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 84df5cae824e47cd85ad2ce1845f935d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682326742700 2023-04-24 08:59:04,668 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting eb15d316bf314034b2f6d692f63e29c8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682326740558 2023-04-24 08:59:04,669 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 2981b8096c4043359f5a7e64b1ab3b94, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682326742696 2023-04-24 08:59:04,673 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,673 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:04,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,678 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#B#compaction#138 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:04,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:04,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:04,679 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:04,679 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:04,679 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:04,679 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:04,679 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:04,686 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#A#compaction#139 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:04,689 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,689 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:04,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,689 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:04,690 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:04,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:04,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:04,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,699 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:04,699 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:04,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:04,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:04,713 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/7a86ee6dbe4d41ba99bb9498376e98ec as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/7a86ee6dbe4d41ba99bb9498376e98ec 2023-04-24 08:59:04,719 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/B of 415f076aa60cbee3cb48b662f1bba024 into 7a86ee6dbe4d41ba99bb9498376e98ec(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:04,720 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:04,720 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/B, priority=11, startTime=1682326744663; duration=0sec 2023-04-24 08:59:04,720 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:04,720 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:59:04,722 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:59:04,722 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/C is initiating minor compaction (all files) 2023-04-24 08:59:04,722 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/C in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,722 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a0f3c024073d4944be26b5f4cf95978a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/431baf35819c4de1b4721f1bbd725af2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3a1b7c9c588f437c8ab65d02591d3817, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/eecaeb0e96ec4d8aaae784853794827c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/aa2c1c81a09b48acb830684d990bee23] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=59.0 K 2023-04-24 08:59:04,722 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a0f3c024073d4944be26b5f4cf95978a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1682326737728 2023-04-24 08:59:04,722 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 431baf35819c4de1b4721f1bbd725af2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682326738270 2023-04-24 08:59:04,723 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3a1b7c9c588f437c8ab65d02591d3817, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1682326739400 2023-04-24 08:59:04,723 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting eecaeb0e96ec4d8aaae784853794827c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682326740558 2023-04-24 08:59:04,723 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting aa2c1c81a09b48acb830684d990bee23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682326742700 2023-04-24 08:59:04,731 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#C#compaction#140 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:04,742 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/a27b0b89de3f417887cae4961cc37ddf as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a27b0b89de3f417887cae4961cc37ddf 2023-04-24 08:59:04,747 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/C of 415f076aa60cbee3cb48b662f1bba024 into a27b0b89de3f417887cae4961cc37ddf(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:04,747 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:04,747 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/C, priority=11, startTime=1682326744664; duration=0sec 2023-04-24 08:59:04,747 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:04,799 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:04,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:04,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:04,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:04,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:04,802 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:04,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:04,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:04,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,807 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:04,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:04,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:04,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:04,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:04,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:04,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:04,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:04,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:04,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:04,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:04,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:04,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:04,808 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:04,808 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:04,809 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 08:59:04,809 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:04,809 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:04,809 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:04,809 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:04,809 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:04,809 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:04,823 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304246574c3fac38e40099d63b08fde17cacc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246574c3fac38e40099d63b08fde17cacc 2023-04-24 08:59:05,125 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/fd2d0311be40423d8ea93439e65bf942 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fd2d0311be40423d8ea93439e65bf942 2023-04-24 08:59:05,129 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/A of 415f076aa60cbee3cb48b662f1bba024 into fd2d0311be40423d8ea93439e65bf942(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:05,129 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:05,129 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/A, priority=11, startTime=1682326744663; duration=0sec 2023-04-24 08:59:05,130 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:05,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:05,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:05,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326805156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326805158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45218 deadline: 1682326805158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,228 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=277, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/f67950942941402caab8deb5b29b950e 2023-04-24 08:59:05,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326805259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326805260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326805260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326805461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326805463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326805463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,639 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/32f8216eb46947d3b54b177059a19a49 2023-04-24 08:59:05,648 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/877b9a2a538f4cb19f6da179f8dc4682 2023-04-24 08:59:05,652 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/f67950942941402caab8deb5b29b950e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f67950942941402caab8deb5b29b950e 2023-04-24 08:59:05,655 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f67950942941402caab8deb5b29b950e, entries=150, sequenceid=277, filesize=25.4 K 2023-04-24 08:59:05,656 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/32f8216eb46947d3b54b177059a19a49 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/32f8216eb46947d3b54b177059a19a49 2023-04-24 08:59:05,660 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/32f8216eb46947d3b54b177059a19a49, entries=150, sequenceid=277, filesize=11.9 K 2023-04-24 08:59:05,661 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/877b9a2a538f4cb19f6da179f8dc4682 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/877b9a2a538f4cb19f6da179f8dc4682 2023-04-24 08:59:05,670 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/877b9a2a538f4cb19f6da179f8dc4682, entries=150, sequenceid=277, filesize=11.9 K 2023-04-24 08:59:05,671 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 415f076aa60cbee3cb48b662f1bba024 in 862ms, sequenceid=277, compaction requested=false 2023-04-24 08:59:05,671 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:05,671 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:05,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:05,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:05,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:05,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:05,706 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:05,707 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:05,707 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:05,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:05,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0d8a21[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:05,708 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:05,708 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,715 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,715 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:05,715 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:05,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:05,715 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:05,723 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:05,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:05,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:05,724 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:05,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:05,725 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:05,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:05,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:05,726 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:05,726 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:05,732 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,732 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:05,732 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:05,732 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,733 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:05,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:05,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:05,740 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:05,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:05,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:05,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:05,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:05,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,741 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,741 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,741 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,741 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:05,741 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:05,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:05,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:05,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:05,765 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 08:59:05,766 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:05,766 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:05,766 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:05,766 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:05,766 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:05,766 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:05,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326805779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326805781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326805781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,841 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:05,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:05,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:05,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:05,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:05,843 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:05,843 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:05,843 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:05,844 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:05,844 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,844 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:05,848 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:05,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:05,849 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,849 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:05,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:05,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:05,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:05,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:05,850 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:05,850 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:05,850 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:05,850 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:05,850 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:05,857 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,857 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:05,857 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,857 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:05,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:05,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3737aa1e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:05,858 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:05,858 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,865 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,865 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,865 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:05,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:05,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:05,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:05,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:05,873 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:05,874 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:05,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:05,875 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:05,875 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:05,875 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:05,875 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:05,875 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:05,882 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,882 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,882 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:05,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,882 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:05,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:05,883 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:05,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45236 deadline: 1682326805882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326805882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326805883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,886 DEBUG [Thread-782] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:59:05,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:05,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326805883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,890 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:05,890 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:05,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:05,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:05,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:05,894 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:05,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:05,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:05,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:05,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:05,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:05,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:05,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,894 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:05,894 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:05,994 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:05,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:05,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:05,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:05,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:05,996 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:05,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:05,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:05,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:05,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:05,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,026 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:06,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:06,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:06,027 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:06,027 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:06,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:06,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:06,028 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,028 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:06,028 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:06,028 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:06,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:06,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:06,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:06,040 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,040 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:06,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff4ca90[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:06,042 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:06,042 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,048 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,049 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:06,049 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,049 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:06,049 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:06,049 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:06,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:06,057 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:06,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:06,059 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:06,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:06,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:06,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:06,059 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:06,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,066 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,073 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,073 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,073 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,074 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:06,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:06,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,074 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:06,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,074 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:06,074 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326806087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326806087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326806091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,174 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:06,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:06,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:06,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:06,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:06,178 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:06,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:06,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:06,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,179 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424112cf40f618948cea5fbb8a256ed3474 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424112cf40f618948cea5fbb8a256ed3474 2023-04-24 08:59:06,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,182 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,182 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,182 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:06,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:06,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:06,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:06,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:06,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:06,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:06,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:06,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:06,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:06,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:06,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:06,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:06,190 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:06,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:06,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fa17b09[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:06,192 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,198 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,199 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:06,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:06,199 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:06,199 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:06,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:06,207 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:06,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:06,209 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:06,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:06,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:06,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:06,209 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:06,215 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,215 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,215 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,216 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,223 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,223 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,223 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,223 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,224 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,224 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,224 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:06,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:06,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,223 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:06,224 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:06,224 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,224 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:06,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,324 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:06,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:06,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:06,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:06,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:06,328 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:06,328 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,329 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:06,329 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:06,329 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,329 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,332 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,332 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,332 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,332 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:06,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:06,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:06,333 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,333 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:06,333 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:06,333 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:06,340 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,341 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,341 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:06,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8eb05e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:06,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:06,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,348 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,349 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:06,349 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:06,349 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,349 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:06,349 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:06,357 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:06,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,357 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,357 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:06,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:06,359 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:06,359 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:06,359 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:06,359 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:06,359 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:06,365 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,365 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,365 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,373 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,373 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,374 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,374 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:06,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,373 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,374 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:06,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:06,374 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,374 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,374 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:06,374 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,374 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:06,374 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326806390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326806390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326806395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,474 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:06,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:06,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:06,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:06,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:06,477 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:06,477 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,477 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:06,477 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:06,478 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,478 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,482 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,482 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,482 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,482 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:06,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:06,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:06,483 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,483 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:06,483 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:06,483 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:06,483 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:06,490 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:06,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e3968b[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:06,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:06,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,518 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,518 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,518 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:06,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:06,518 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:06,519 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:06,532 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:06,532 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,532 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:06,532 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:06,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:06,534 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:06,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:06,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:06,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:06,534 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:06,540 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,540 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,540 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,541 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,542 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,549 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:06,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:06,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:06,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,549 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,549 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:06,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,549 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,585 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=292, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/c504acddaa804248bf6151dcdf21d5bf 2023-04-24 08:59:06,649 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:06,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:06,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:06,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:06,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:06,652 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:06,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:06,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:06,653 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,653 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,657 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:06,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:06,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:06,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:06,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:06,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:06,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:06,658 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,658 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:06,658 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:06,658 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:06,658 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:06,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:06,659 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:06,665 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,665 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,665 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,665 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:06,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f7a668d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:06,667 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:06,667 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,673 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:06,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:06,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:06,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:06,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:06,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,682 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:06,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:06,684 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:06,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:06,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:06,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:06,684 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:06,690 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,690 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,690 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,692 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,698 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,698 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:06,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:06,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:06,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,699 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,699 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:06,699 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:06,799 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:06,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:06,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:06,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:06,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:06,802 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:06,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:06,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:06,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,807 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:06,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:06,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:06,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:06,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:06,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:06,808 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:06,808 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,808 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:06,808 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:06,808 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,809 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:06,809 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:06,809 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,809 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:06,809 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:06,815 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,815 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,815 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,816 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:06,817 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:06,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ba73158[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:06,817 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,823 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,824 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:06,824 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,824 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:06,824 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:06,824 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:06,832 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,832 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,832 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,832 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:06,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:06,834 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:06,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:06,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:06,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:06,834 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:06,840 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,840 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,840 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,840 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,841 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,848 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,848 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:06,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:06,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:06,848 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,849 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,849 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,849 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:06,849 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:06,849 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:06,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:06,849 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,849 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,849 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326806893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326806893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326806899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,949 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:06,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:06,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:06,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:06,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:06,952 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:06,952 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,952 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:06,952 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:06,952 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,952 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:06,957 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:06,957 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,957 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:06,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:06,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:06,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:06,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:06,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:06,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:06,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:06,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:06,958 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:06,958 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,959 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:06,959 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:06,959 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:06,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:06,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:06,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:06,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:06,965 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,965 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:06,965 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:06,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:06,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:06,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62027432[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:06,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,973 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:06,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:06,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:06,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:06,974 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:06,974 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:06,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/922dafc4e5c541c8be08ac242129e4d0 2023-04-24 08:59:07,017 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,017 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,017 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:07,017 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:07,019 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:07,020 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:07,020 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:07,020 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:07,020 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:07,039 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,039 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,039 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,048 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:07,048 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:07,049 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:07,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:07,048 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:07,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:07,049 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,049 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:07,049 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:07,049 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:07,049 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,049 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,049 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:07,049 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:07,149 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:07,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:07,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:07,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:07,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:07,152 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:07,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:07,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:07,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:07,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,157 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:07,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:07,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:07,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:07,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:07,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:07,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:07,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:07,158 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:07,158 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:07,158 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:07,158 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:07,158 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:07,158 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,159 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:07,159 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:07,159 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,159 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:07,159 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:07,159 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,159 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:07,159 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:07,166 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:07,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@289ff137[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:07,169 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:07,169 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,174 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,174 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,174 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:07,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:07,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:07,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:07,182 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:07,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,182 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,182 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:07,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:07,184 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:07,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:07,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:07,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:07,184 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:07,190 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,190 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:07,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,190 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,192 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,198 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,198 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:07,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:07,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:07,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:07,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:07,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:07,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:07,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,199 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:07,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:07,199 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,199 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:07,299 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:07,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:07,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:07,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:07,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:07,302 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:07,302 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:07,302 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:07,302 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:07,303 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,303 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,307 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:07,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:07,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:07,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:07,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:07,308 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,308 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:07,308 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:07,308 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:07,308 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:07,315 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,315 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,315 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,315 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:07,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:07,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14edb4a7[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:07,316 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:07,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,323 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,324 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,324 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:07,330 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:07,331 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:07,331 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:07,340 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:07,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:07,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:07,342 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:07,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:07,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:07,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:07,343 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:07,348 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,348 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,349 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,349 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,350 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:07,357 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:07,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:07,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:07,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,357 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:07,357 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:07,357 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,404 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/184a4652671743efaebe8183e17ad936 2023-04-24 08:59:07,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/c504acddaa804248bf6151dcdf21d5bf as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c504acddaa804248bf6151dcdf21d5bf 2023-04-24 08:59:07,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c504acddaa804248bf6151dcdf21d5bf, entries=200, sequenceid=292, filesize=32.3 K 2023-04-24 08:59:07,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/922dafc4e5c541c8be08ac242129e4d0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/922dafc4e5c541c8be08ac242129e4d0 2023-04-24 08:59:07,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/922dafc4e5c541c8be08ac242129e4d0, entries=150, sequenceid=292, filesize=11.9 K 2023-04-24 08:59:07,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/184a4652671743efaebe8183e17ad936 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/184a4652671743efaebe8183e17ad936 2023-04-24 08:59:07,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/184a4652671743efaebe8183e17ad936, entries=150, sequenceid=292, filesize=11.9 K 2023-04-24 08:59:07,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 415f076aa60cbee3cb48b662f1bba024 in 1658ms, sequenceid=292, compaction requested=true 2023-04-24 08:59:07,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:07,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:07,423 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:07,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:07,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-24 08:59:07,425 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85472 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:07,425 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:07,425 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/A is initiating minor compaction (all files) 2023-04-24 08:59:07,425 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/A in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,425 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fd2d0311be40423d8ea93439e65bf942, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f67950942941402caab8deb5b29b950e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c504acddaa804248bf6151dcdf21d5bf] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=83.5 K 2023-04-24 08:59:07,426 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting fd2d0311be40423d8ea93439e65bf942, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682326742700 2023-04-24 08:59:07,426 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:07,426 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/B is initiating minor compaction (all files) 2023-04-24 08:59:07,426 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/B in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,426 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/7a86ee6dbe4d41ba99bb9498376e98ec, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/32f8216eb46947d3b54b177059a19a49, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/922dafc4e5c541c8be08ac242129e4d0] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=36.0 K 2023-04-24 08:59:07,426 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f67950942941402caab8deb5b29b950e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682326743022 2023-04-24 08:59:07,426 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7a86ee6dbe4d41ba99bb9498376e98ec, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682326742700 2023-04-24 08:59:07,426 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c504acddaa804248bf6151dcdf21d5bf, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682326745153 2023-04-24 08:59:07,427 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 32f8216eb46947d3b54b177059a19a49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682326743022 2023-04-24 08:59:07,427 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 922dafc4e5c541c8be08ac242129e4d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682326745157 2023-04-24 08:59:07,436 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#B#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:07,452 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#A#compaction#148 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:07,457 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:07,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:07,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:07,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:07,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:07,461 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:07,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:07,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:07,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:07,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:07,465 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:07,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:07,465 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,466 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:07,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:07,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:07,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:07,467 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:07,467 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:07,467 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:07,471 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:07,472 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,472 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:07,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:07,472 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 08:59:07,472 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:07,472 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:07,472 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:07,472 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:07,472 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:07,472 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:07,497 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304246bd657920fe0444ba697710a519af4d9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246bd657920fe0444ba697710a519af4d9 2023-04-24 08:59:07,858 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/37d285d7a9dc4bb8ae179e7b7c7fbbe7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/37d285d7a9dc4bb8ae179e7b7c7fbbe7 2023-04-24 08:59:07,863 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/B of 415f076aa60cbee3cb48b662f1bba024 into 37d285d7a9dc4bb8ae179e7b7c7fbbe7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:07,863 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:07,863 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/B, priority=13, startTime=1682326747423; duration=0sec 2023-04-24 08:59:07,863 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:07,863 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:07,865 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:07,866 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/C is initiating minor compaction (all files) 2023-04-24 08:59:07,866 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/C in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,866 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a27b0b89de3f417887cae4961cc37ddf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/877b9a2a538f4cb19f6da179f8dc4682, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/184a4652671743efaebe8183e17ad936] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=36.0 K 2023-04-24 08:59:07,867 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a27b0b89de3f417887cae4961cc37ddf, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682326742700 2023-04-24 08:59:07,868 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 877b9a2a538f4cb19f6da179f8dc4682, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1682326743022 2023-04-24 08:59:07,868 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 184a4652671743efaebe8183e17ad936, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682326745157 2023-04-24 08:59:07,877 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#C#compaction#150 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:07,889 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/56d8ab724b7f4546a965b7c91c64d5dd as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/56d8ab724b7f4546a965b7c91c64d5dd 2023-04-24 08:59:07,895 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/C of 415f076aa60cbee3cb48b662f1bba024 into 56d8ab724b7f4546a965b7c91c64d5dd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:07,896 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:07,896 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/C, priority=13, startTime=1682326747424; duration=0sec 2023-04-24 08:59:07,896 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:07,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:07,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:07,903 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=310, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/fecb8ed56cbb41bcb6da78c38e7ca4df 2023-04-24 08:59:07,916 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/ff3cc6c892534caf80016c5c11fa419c 2023-04-24 08:59:07,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:07,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326807917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:07,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:07,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326807921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326807921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,930 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/a1e06a68f54d49da94cbbba5e65dc44a 2023-04-24 08:59:07,939 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/fecb8ed56cbb41bcb6da78c38e7ca4df as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fecb8ed56cbb41bcb6da78c38e7ca4df 2023-04-24 08:59:07,944 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fecb8ed56cbb41bcb6da78c38e7ca4df, entries=100, sequenceid=310, filesize=18.6 K 2023-04-24 08:59:07,946 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/ff3cc6c892534caf80016c5c11fa419c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/ff3cc6c892534caf80016c5c11fa419c 2023-04-24 08:59:07,950 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/ff3cc6c892534caf80016c5c11fa419c, entries=100, sequenceid=310, filesize=9.5 K 2023-04-24 08:59:07,951 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/a1e06a68f54d49da94cbbba5e65dc44a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a1e06a68f54d49da94cbbba5e65dc44a 2023-04-24 08:59:07,957 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a1e06a68f54d49da94cbbba5e65dc44a, entries=100, sequenceid=310, filesize=9.5 K 2023-04-24 08:59:07,958 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 415f076aa60cbee3cb48b662f1bba024 in 486ms, sequenceid=310, compaction requested=false 2023-04-24 08:59:07,959 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:07,959 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:07,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:07,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:07,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:07,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:07,965 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,965 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,966 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b26a93[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:07,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:07,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,973 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:07,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:07,974 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,974 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:07,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:07,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:07,982 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:07,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:07,982 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:07,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:07,984 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:07,984 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:07,984 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:07,984 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:07,984 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:08,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:08,025 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-24 08:59:08,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:08,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:08,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:08,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:08,025 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:08,025 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:08,040 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,040 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,040 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,043 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326808045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326808047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326808047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,070 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:08,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:08,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:08,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,070 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,071 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:08,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,071 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:08,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326808148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326808151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326808152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,172 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:08,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:08,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:08,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:08,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:08,174 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:08,174 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:08,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:08,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,175 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,182 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,182 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,182 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,182 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:08,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:08,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:08,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:08,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:08,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,183 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:08,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:08,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,190 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:08,192 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:08,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17e02193[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:08,192 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,198 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,199 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:08,199 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:08,199 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:08,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:08,207 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:08,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:08,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:08,209 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:08,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:08,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:08,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:08,209 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:08,215 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,215 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,215 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,216 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,258 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,258 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:08,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:08,258 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,258 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,259 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,259 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,259 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,259 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:08,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,259 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,259 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:08,259 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326808350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326808354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326808355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,359 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:08,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:08,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:08,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:08,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:08,361 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:08,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:08,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:08,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,381 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:08,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:08,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:08,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:08,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:08,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:08,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:08,382 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,383 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:08,383 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:08,383 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:08,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:08,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:08,383 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:08,390 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,391 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:08,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:08,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c03b37d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:08,392 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:08,392 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,399 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,399 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,399 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:08,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,399 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:08,399 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:08,399 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:08,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:08,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,407 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:08,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:08,409 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:08,409 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:08,409 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:08,409 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:08,409 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:08,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,415 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,416 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,417 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,424 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,424 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,424 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,424 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:08,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,424 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,424 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,424 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,424 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,424 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,424 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:08,424 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:08,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:08,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,445 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424d698e93d6f624d1c95a762139e664df1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d698e93d6f624d1c95a762139e664df1 2023-04-24 08:59:08,457 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=327, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/c414b54c719148c2a825f779d409be95 2023-04-24 08:59:08,524 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:08,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:08,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:08,527 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:08,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:08,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:08,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,532 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,532 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,532 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,532 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:08,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:08,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:08,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:08,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:08,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:08,533 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:08,533 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,533 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:08,533 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:08,533 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,534 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:08,534 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:08,534 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,534 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:08,534 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:08,540 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:08,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a79ffd7[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:08,542 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:08,542 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,549 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,549 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:08,549 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:08,549 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:08,549 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:08,557 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:08,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:08,559 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:08,559 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:08,559 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:08,559 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:08,559 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:08,565 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,565 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,565 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,566 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,567 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,596 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:08,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:08,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,596 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:08,596 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326808652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326808657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:08,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326808658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,683 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/f44d1ba1b3fb4f97869f709553784f9b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f44d1ba1b3fb4f97869f709553784f9b 2023-04-24 08:59:08,689 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/A of 415f076aa60cbee3cb48b662f1bba024 into f44d1ba1b3fb4f97869f709553784f9b(size=26.0 K), total size for store is 44.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-24 08:59:08,689 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:08,689 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/A, priority=13, startTime=1682326747423; duration=1sec 2023-04-24 08:59:08,689 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:08,696 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:08,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:08,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:08,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:08,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:08,699 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:08,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:08,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:08,700 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,700 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,711 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,711 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,712 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,712 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:08,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:08,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:08,713 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,713 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:08,713 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:08,713 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:08,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:08,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:08,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1173516d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:08,721 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:08,721 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,728 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,728 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:08,728 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,728 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:08,728 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:08,728 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:08,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:08,737 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,737 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,737 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:08,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:08,739 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:08,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:08,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:08,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:08,739 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:08,745 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,745 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,746 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,753 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,753 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:08,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:08,753 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,753 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,753 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,753 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,753 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:08,753 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,753 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:08,754 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:08,760 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,760 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,760 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:08,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,854 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:08,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:08,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:08,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:08,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:08,857 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:08,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:08,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:08,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,861 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,861 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,862 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,862 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:08,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:08,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:08,863 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,863 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:08,863 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:08,863 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:08,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:08,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/e24e0839ab57486c8a14625a753bd683 2023-04-24 08:59:08,870 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:08,871 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:08,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dde122d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:08,871 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/85c8eb9e6bed4ed28c55445488721fb7 2023-04-24 08:59:08,878 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,878 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:08,878 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:08,878 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,878 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:08,878 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:08,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/c414b54c719148c2a825f779d409be95 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c414b54c719148c2a825f779d409be95 2023-04-24 08:59:08,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c414b54c719148c2a825f779d409be95, entries=200, sequenceid=327, filesize=32.3 K 2023-04-24 08:59:08,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/e24e0839ab57486c8a14625a753bd683 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/e24e0839ab57486c8a14625a753bd683 2023-04-24 08:59:08,886 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:08,886 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,887 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,887 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:08,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:08,889 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:08,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:08,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:08,889 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/e24e0839ab57486c8a14625a753bd683, entries=150, sequenceid=327, filesize=11.9 K 2023-04-24 08:59:08,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:08,889 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:08,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/85c8eb9e6bed4ed28c55445488721fb7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/85c8eb9e6bed4ed28c55445488721fb7 2023-04-24 08:59:08,895 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,895 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,895 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:08,895 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:08,896 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/85c8eb9e6bed4ed28c55445488721fb7, entries=150, sequenceid=327, filesize=11.9 K 2023-04-24 08:59:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:08,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 415f076aa60cbee3cb48b662f1bba024 in 873ms, sequenceid=327, compaction requested=true 2023-04-24 08:59:08,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:08,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:08,898 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:08,898 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:08,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:08,898 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:08,899 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78676 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:08,899 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:08,899 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/A is initiating minor compaction (all files) 2023-04-24 08:59:08,899 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/B is initiating minor compaction (all files) 2023-04-24 08:59:08,899 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/A in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,899 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/B in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:08,899 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f44d1ba1b3fb4f97869f709553784f9b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fecb8ed56cbb41bcb6da78c38e7ca4df, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c414b54c719148c2a825f779d409be95] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=76.8 K 2023-04-24 08:59:08,899 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/37d285d7a9dc4bb8ae179e7b7c7fbbe7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/ff3cc6c892534caf80016c5c11fa419c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/e24e0839ab57486c8a14625a753bd683] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=33.8 K 2023-04-24 08:59:08,900 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f44d1ba1b3fb4f97869f709553784f9b, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682326745157 2023-04-24 08:59:08,900 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 37d285d7a9dc4bb8ae179e7b7c7fbbe7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682326745157 2023-04-24 08:59:08,900 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting fecb8ed56cbb41bcb6da78c38e7ca4df, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1682326745780 2023-04-24 08:59:08,900 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting ff3cc6c892534caf80016c5c11fa419c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1682326745780 2023-04-24 08:59:08,900 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c414b54c719148c2a825f779d409be95, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682326747915 2023-04-24 08:59:08,901 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e24e0839ab57486c8a14625a753bd683, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682326747915 2023-04-24 08:59:08,903 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,903 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:08,903 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:08,903 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:08,903 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:08,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:08,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:08,903 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:08,903 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:08,904 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,904 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:08,904 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:08,904 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:08,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:08,904 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:08,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:08,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:08,907 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#B#compaction#157 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:08,913 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#A#compaction#156 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:09,004 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:09,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:09,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:09,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:09,006 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 08:59:09,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:09,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:09,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:09,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:09,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:09,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:09,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:09,007 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:09,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:09,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:09,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:09,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,011 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:09,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:09,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:09,013 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:09,013 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,013 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:09,013 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:09,014 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:09,014 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:09,014 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:09,014 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:09,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:09,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:09,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:09,020 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c97c4b0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:09,023 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:09,023 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,028 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:09,028 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,028 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:09,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:09,029 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:09,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:09,036 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,037 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,037 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:09,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:09,038 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:09,038 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:09,038 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:09,038 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:09,038 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:09,045 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,045 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,045 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,046 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,047 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326809045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,053 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,053 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,053 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,053 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:09,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:09,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:09,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,053 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,054 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:09,054 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,054 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:09,054 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,054 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,054 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,054 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:09,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326809148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,154 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:09,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:09,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:09,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:09,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:09,156 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:09,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:09,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:09,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:09,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326809157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:09,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326809160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:09,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326809161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,180 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,180 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,180 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,180 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:09,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:09,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:09,181 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,181 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:09,181 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:09,181 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:09,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:09,194 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,195 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,195 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:09,196 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:09,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69850bb2[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:09,196 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,203 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,203 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,203 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:09,203 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:09,203 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:09,203 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:09,211 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:09,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,211 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,212 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:09,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:09,213 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:09,213 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:09,213 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:09,213 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:09,213 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:09,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,220 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,228 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,228 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,228 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:09,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:09,228 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:09,228 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,228 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,229 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,229 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:09,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,229 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:09,229 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,229 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,229 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,328 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/339390390c07421ebb928dac7d256122 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/339390390c07421ebb928dac7d256122 2023-04-24 08:59:09,329 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:09,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:09,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:09,333 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/B of 415f076aa60cbee3cb48b662f1bba024 into 339390390c07421ebb928dac7d256122(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:09,334 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:09,334 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/B, priority=13, startTime=1682326748898; duration=0sec 2023-04-24 08:59:09,334 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:09,334 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:09,335 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:09,335 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/C is initiating minor compaction (all files) 2023-04-24 08:59:09,335 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/C in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,335 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/56d8ab724b7f4546a965b7c91c64d5dd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a1e06a68f54d49da94cbbba5e65dc44a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/85c8eb9e6bed4ed28c55445488721fb7] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=33.8 K 2023-04-24 08:59:09,336 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 56d8ab724b7f4546a965b7c91c64d5dd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682326745157 2023-04-24 08:59:09,336 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a1e06a68f54d49da94cbbba5e65dc44a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1682326745780 2023-04-24 08:59:09,336 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 85c8eb9e6bed4ed28c55445488721fb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682326747915 2023-04-24 08:59:09,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:09,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:09,339 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:09,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:09,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:09,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,340 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,342 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#C#compaction#159 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:09,345 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,345 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,345 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:09,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:09,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:09,346 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,346 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:09,346 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:09,346 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:09,346 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:09,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:09,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326809351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:09,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62f6d7d6[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:09,355 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:09,355 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,362 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:09,362 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:09,362 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:09,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:09,370 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:09,370 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,370 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,370 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:09,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:09,372 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:09,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:09,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:09,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:09,372 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:09,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,386 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:09,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:09,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:09,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,387 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:09,387 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:09,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,387 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,387 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,387 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,426 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042462990b03d7f6480aae2b8a3f61abccf7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042462990b03d7f6480aae2b8a3f61abccf7 2023-04-24 08:59:09,431 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=346, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/ea15131db3a746049c3c3e814fce4194 2023-04-24 08:59:09,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/519c9a3200f04c3a96e733f5a5fe70f7 2023-04-24 08:59:09,487 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:09,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:09,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:09,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:09,490 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:09,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:09,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:09,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,495 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,495 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,495 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,495 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:09,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:09,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:09,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:09,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:09,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:09,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:09,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,504 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:09,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fe4eedf[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:09,504 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:09,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,511 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,512 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:09,512 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:09,512 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:09,512 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,512 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:09,520 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:09,520 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,520 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:09,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:09,522 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:09,522 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:09,522 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:09,522 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:09,522 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:09,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,529 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,530 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,537 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,537 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:09,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:09,537 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:09,537 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,537 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:09,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,537 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:09,537 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,537 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,537 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,537 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,637 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:09,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:09,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:09,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:09,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:09,640 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:09,640 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,640 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:09,640 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:09,640 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,640 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,645 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:09,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:09,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:09,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:09,646 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:09,646 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,646 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:09,646 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:09,646 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:09,646 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,646 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:09,646 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:09,646 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,646 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:09,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:09,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:09,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:09,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326809654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,672 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,673 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:09,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d941fd7[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:09,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:09,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:09,687 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,687 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:09,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:09,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:09,695 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,695 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:09,695 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,695 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:09,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:09,697 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:09,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:09,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:09,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:09,697 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:09,703 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,703 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,703 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,704 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,705 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,711 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,711 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,712 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,712 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:09,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:09,712 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,712 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,712 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,712 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,712 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:09,712 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:09,712 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:09,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,712 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,750 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/e546fee51d314992847c782c0a46c2a7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/e546fee51d314992847c782c0a46c2a7 2023-04-24 08:59:09,753 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/C of 415f076aa60cbee3cb48b662f1bba024 into e546fee51d314992847c782c0a46c2a7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:09,753 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:09,753 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/C, priority=13, startTime=1682326748898; duration=0sec 2023-04-24 08:59:09,754 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:09,812 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:09,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:09,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:09,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:09,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:09,815 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:09,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:09,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:09,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,820 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:09,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:09,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:09,821 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:09,821 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,821 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:09,821 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:09,821 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:09,822 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,822 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:09,822 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:09,822 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,822 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:09,822 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:09,822 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,822 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:09,822 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:09,828 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,828 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:09,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c8d434c[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:09,830 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:09,830 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,837 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:09,837 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:09,837 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:09,837 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,837 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:09,845 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:09,845 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,845 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,845 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:09,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:09,847 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:09,847 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:09,847 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:09,847 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:09,847 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:09,850 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/3cb478c9a404440c90815401cbf3d9d7 2023-04-24 08:59:09,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:09,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,854 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/ea15131db3a746049c3c3e814fce4194 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/ea15131db3a746049c3c3e814fce4194 2023-04-24 08:59:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:09,855 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,868 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:09,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:09,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/ea15131db3a746049c3c3e814fce4194, entries=200, sequenceid=346, filesize=32.3 K 2023-04-24 08:59:09,868 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:09,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:09,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,868 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:09,869 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:09,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:09,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:09,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/519c9a3200f04c3a96e733f5a5fe70f7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/519c9a3200f04c3a96e733f5a5fe70f7 2023-04-24 08:59:09,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/519c9a3200f04c3a96e733f5a5fe70f7, entries=150, sequenceid=346, filesize=11.9 K 2023-04-24 08:59:09,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/3cb478c9a404440c90815401cbf3d9d7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3cb478c9a404440c90815401cbf3d9d7 2023-04-24 08:59:09,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3cb478c9a404440c90815401cbf3d9d7, entries=150, sequenceid=346, filesize=11.9 K 2023-04-24 08:59:09,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 415f076aa60cbee3cb48b662f1bba024 in 880ms, sequenceid=346, compaction requested=false 2023-04-24 08:59:09,886 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:09,969 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:09,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:09,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:09,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:09,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:09,971 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:09,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:09,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:09,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,978 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:09,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:09,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:09,979 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:09,979 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:09,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:09,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:09,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:09,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:09,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:09,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:09,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:09,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:09,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:09,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. started... 2023-04-24 08:59:09,980 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 08:59:09,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:09,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:09,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:09,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:09,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:09,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:09,994 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042442f9837723134d7e853a9ada60714c9a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042442f9837723134d7e853a9ada60714c9a 2023-04-24 08:59:10,001 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/5d2739ce6f3743cfa742ab68773337ac 2023-04-24 08:59:10,136 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/583851d1df4b42dbbee3517a8d0c8fb9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/583851d1df4b42dbbee3517a8d0c8fb9 2023-04-24 08:59:10,141 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/A of 415f076aa60cbee3cb48b662f1bba024 into 583851d1df4b42dbbee3517a8d0c8fb9(size=26.1 K), total size for store is 58.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-24 08:59:10,141 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:10,141 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/A, priority=13, startTime=1682326748898; duration=1sec 2023-04-24 08:59:10,141 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:10,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:10,162 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. as already flushing 2023-04-24 08:59:10,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326810199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326810200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326810200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326810201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326810307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326810307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326810307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326810307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,412 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/4ce397f44f7644ca943a3c89e53a887a 2023-04-24 08:59:10,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326810509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326810510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326810510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326810510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,644 DEBUG [Thread-791] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b669b1 to 127.0.0.1:63978 2023-04-24 08:59:10,644 DEBUG [Thread-791] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:10,646 DEBUG [Thread-797] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4890ce53 to 127.0.0.1:63978 2023-04-24 08:59:10,646 DEBUG [Thread-797] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:10,646 DEBUG [Thread-795] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x615d8f34 to 127.0.0.1:63978 2023-04-24 08:59:10,646 DEBUG [Thread-795] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:10,647 DEBUG [Thread-793] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6683e622 to 127.0.0.1:63978 2023-04-24 08:59:10,647 DEBUG [Thread-793] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:10,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:45202 deadline: 1682326810811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45182 deadline: 1682326810813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45208 deadline: 1682326810815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:10,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:45218 deadline: 1682326810817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,822 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/8b882626a8e6481cab959b1b7d647b29 2023-04-24 08:59:10,830 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/5d2739ce6f3743cfa742ab68773337ac as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/5d2739ce6f3743cfa742ab68773337ac 2023-04-24 08:59:10,835 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/5d2739ce6f3743cfa742ab68773337ac, entries=150, sequenceid=362, filesize=25.4 K 2023-04-24 08:59:10,836 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/4ce397f44f7644ca943a3c89e53a887a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/4ce397f44f7644ca943a3c89e53a887a 2023-04-24 08:59:10,841 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/4ce397f44f7644ca943a3c89e53a887a, entries=150, sequenceid=362, filesize=11.9 K 2023-04-24 08:59:10,842 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/8b882626a8e6481cab959b1b7d647b29 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8b882626a8e6481cab959b1b7d647b29 2023-04-24 08:59:10,848 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8b882626a8e6481cab959b1b7d647b29, entries=150, sequenceid=362, filesize=11.9 K 2023-04-24 08:59:10,848 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 415f076aa60cbee3cb48b662f1bba024 in 868ms, sequenceid=362, compaction requested=true 2023-04-24 08:59:10,849 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:10,849 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:10,849 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:10,849 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:10,849 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,849 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:10,849 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:10,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,853 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:10,854 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:10,854 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:10,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:10,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68116b6c[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:10,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:10,857 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:10,862 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:10,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:10,862 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:10,862 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:10,862 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:10,862 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,862 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:10,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:10,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:10,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:10,870 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:10,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:10,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:10,873 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:10,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:10,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:10,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:10,873 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:10,878 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:10,878 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:10,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:10,878 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:10,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:10,879 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,881 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:10,887 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:10,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:10,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:10,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:10,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:10,887 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:10,888 DEBUG [Thread-790] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:10,888 DEBUG [Thread-790] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:10,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:10,988 DEBUG [Thread-790] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:10,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:11,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:11,318 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 08:59:11,318 DEBUG [Thread-778] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dc16927 to 127.0.0.1:63978 2023-04-24 08:59:11,318 DEBUG [Thread-778] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:11,318 DEBUG [Thread-780] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x029d68f7 to 127.0.0.1:63978 2023-04-24 08:59:11,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:11,318 DEBUG [Thread-780] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:11,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:11,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:11,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:11,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:11,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:11,320 DEBUG [Thread-788] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d07b2f5 to 127.0.0.1:63978 2023-04-24 08:59:11,320 DEBUG [Thread-788] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:11,321 DEBUG [Thread-784] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4af56759 to 127.0.0.1:63978 2023-04-24 08:59:11,321 DEBUG [Thread-784] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:11,736 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424462399f3c1064ef0850a25539bc94c62 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424462399f3c1064ef0850a25539bc94c62 2023-04-24 08:59:12,144 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=383, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/b62fdd6c9c0149d89ea58d0046e13fdc 2023-04-24 08:59:12,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/c4cf0fdde6d74e02b354416e7165044c 2023-04-24 08:59:12,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/dc910eddbabe408996bf0b49bd0815c3 2023-04-24 08:59:12,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/b62fdd6c9c0149d89ea58d0046e13fdc as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/b62fdd6c9c0149d89ea58d0046e13fdc 2023-04-24 08:59:13,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/b62fdd6c9c0149d89ea58d0046e13fdc, entries=150, sequenceid=383, filesize=25.4 K 2023-04-24 08:59:13,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/c4cf0fdde6d74e02b354416e7165044c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c4cf0fdde6d74e02b354416e7165044c 2023-04-24 08:59:13,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c4cf0fdde6d74e02b354416e7165044c, entries=150, sequenceid=383, filesize=11.9 K 2023-04-24 08:59:13,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/dc910eddbabe408996bf0b49bd0815c3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/dc910eddbabe408996bf0b49bd0815c3 2023-04-24 08:59:13,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/dc910eddbabe408996bf0b49bd0815c3, entries=150, sequenceid=383, filesize=11.9 K 2023-04-24 08:59:13,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=13.42 KB/13740 for 415f076aa60cbee3cb48b662f1bba024 in 1696ms, sequenceid=383, compaction requested=true 2023-04-24 08:59:13,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:13,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:13,014 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:13,014 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:13,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:13,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:13,015 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111872 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:13,015 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:13,015 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/A is initiating minor compaction (all files) 2023-04-24 08:59:13,015 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/B is initiating minor compaction (all files) 2023-04-24 08:59:13,015 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/A in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:13,015 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/B in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:13,015 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/583851d1df4b42dbbee3517a8d0c8fb9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/ea15131db3a746049c3c3e814fce4194, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/5d2739ce6f3743cfa742ab68773337ac, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/b62fdd6c9c0149d89ea58d0046e13fdc] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=109.3 K 2023-04-24 08:59:13,015 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/339390390c07421ebb928dac7d256122, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/519c9a3200f04c3a96e733f5a5fe70f7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/4ce397f44f7644ca943a3c89e53a887a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c4cf0fdde6d74e02b354416e7165044c] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=48.2 K 2023-04-24 08:59:13,016 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 339390390c07421ebb928dac7d256122, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682326747915 2023-04-24 08:59:13,016 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 583851d1df4b42dbbee3517a8d0c8fb9, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682326747915 2023-04-24 08:59:13,016 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 519c9a3200f04c3a96e733f5a5fe70f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1682326748044 2023-04-24 08:59:13,016 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting ea15131db3a746049c3c3e814fce4194, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1682326748042 2023-04-24 08:59:13,016 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4ce397f44f7644ca943a3c89e53a887a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682326749026 2023-04-24 08:59:13,016 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 5d2739ce6f3743cfa742ab68773337ac, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682326749026 2023-04-24 08:59:13,017 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c4cf0fdde6d74e02b354416e7165044c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682326750200 2023-04-24 08:59:13,017 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting b62fdd6c9c0149d89ea58d0046e13fdc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682326750200 2023-04-24 08:59:13,025 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#B#compaction#168 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:13,028 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#A#compaction#169 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:13,440 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/5c519b62699642d1bc5626fb48aca679 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/5c519b62699642d1bc5626fb48aca679 2023-04-24 08:59:13,446 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/B of 415f076aa60cbee3cb48b662f1bba024 into 5c519b62699642d1bc5626fb48aca679(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:13,446 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:13,446 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/B, priority=12, startTime=1682326753014; duration=0sec 2023-04-24 08:59:13,446 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:13,446 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:13,448 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:13,448 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 415f076aa60cbee3cb48b662f1bba024/C is initiating minor compaction (all files) 2023-04-24 08:59:13,449 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 415f076aa60cbee3cb48b662f1bba024/C in TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:13,449 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/e546fee51d314992847c782c0a46c2a7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3cb478c9a404440c90815401cbf3d9d7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8b882626a8e6481cab959b1b7d647b29, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/dc910eddbabe408996bf0b49bd0815c3] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp, totalSize=48.2 K 2023-04-24 08:59:13,449 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e546fee51d314992847c782c0a46c2a7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1682326747915 2023-04-24 08:59:13,450 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3cb478c9a404440c90815401cbf3d9d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1682326748044 2023-04-24 08:59:13,450 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 8b882626a8e6481cab959b1b7d647b29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682326749026 2023-04-24 08:59:13,451 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting dc910eddbabe408996bf0b49bd0815c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1682326750200 2023-04-24 08:59:13,463 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 415f076aa60cbee3cb48b662f1bba024#C#compaction#170 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:13,875 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/faaf7e3786fc4d1795825367cd654a00 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/faaf7e3786fc4d1795825367cd654a00 2023-04-24 08:59:13,882 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/C of 415f076aa60cbee3cb48b662f1bba024 into faaf7e3786fc4d1795825367cd654a00(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:13,882 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:13,882 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/C, priority=12, startTime=1682326753014; duration=0sec 2023-04-24 08:59:13,882 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:14,258 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/cd8d71f5215a428f95e293028d3b6a84 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cd8d71f5215a428f95e293028d3b6a84 2023-04-24 08:59:14,263 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 415f076aa60cbee3cb48b662f1bba024/A of 415f076aa60cbee3cb48b662f1bba024 into cd8d71f5215a428f95e293028d3b6a84(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-24 08:59:14,264 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:14,264 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024., storeName=415f076aa60cbee3cb48b662f1bba024/A, priority=12, startTime=1682326753014; duration=1sec 2023-04-24 08:59:14,264 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:15,986 DEBUG [Thread-782] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6322d500 to 127.0.0.1:63978 2023-04-24 08:59:15,987 DEBUG [Thread-782] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:15,987 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 08:59:15,987 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-04-24 08:59:15,988 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-24 08:59:15,988 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-24 08:59:15,988 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-04-24 08:59:15,988 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-24 08:59:15,988 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 08:59:15,988 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5813 2023-04-24 08:59:15,989 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5855 2023-04-24 08:59:15,989 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 08:59:15,989 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2497 2023-04-24 08:59:15,989 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7491 rows 2023-04-24 08:59:15,989 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2494 2023-04-24 08:59:15,989 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7476 rows 2023-04-24 08:59:15,989 INFO [Listener at localhost.localdomain/44291] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 08:59:15,990 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10dd04e6 to 127.0.0.1:63978 2023-04-24 08:59:15,990 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:15,992 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 08:59:15,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-24 08:59:15,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:15,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 08:59:15,997 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326755997"}]},"ts":"1682326755997"} 2023-04-24 08:59:15,998 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 08:59:16,057 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 08:59:16,059 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, UNASSIGN}] 2023-04-24 08:59:16,061 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, UNASSIGN 2023-04-24 08:59:16,063 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=415f076aa60cbee3cb48b662f1bba024, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:16,063 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326756063"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326756063"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326756063"}]},"ts":"1682326756063"} 2023-04-24 08:59:16,067 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:59:16,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 08:59:16,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:16,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 415f076aa60cbee3cb48b662f1bba024, disabling compactions & flushes 2023-04-24 08:59:16,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:16,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:16,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. after waiting 0 ms 2023-04-24 08:59:16,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:16,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 415f076aa60cbee3cb48b662f1bba024 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-24 08:59:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=A 2023-04-24 08:59:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=B 2023-04-24 08:59:16,222 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:16,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 415f076aa60cbee3cb48b662f1bba024, store=C 2023-04-24 08:59:16,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 08:59:16,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 08:59:16,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424093ab23aab564093898c72f005366ea7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424093ab23aab564093898c72f005366ea7 2023-04-24 08:59:17,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=392, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/c6297f2b1fca4aa69b9fd53a21f20f4a 2023-04-24 08:59:17,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 08:59:17,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/6c477336465b460591a56e9e31fa0ce8 2023-04-24 08:59:17,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/7e4aed69f18f46e49bf9ae7539d3730c 2023-04-24 08:59:17,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/A/c6297f2b1fca4aa69b9fd53a21f20f4a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c6297f2b1fca4aa69b9fd53a21f20f4a 2023-04-24 08:59:17,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c6297f2b1fca4aa69b9fd53a21f20f4a, entries=100, sequenceid=392, filesize=18.6 K 2023-04-24 08:59:17,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/B/6c477336465b460591a56e9e31fa0ce8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/6c477336465b460591a56e9e31fa0ce8 2023-04-24 08:59:17,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/6c477336465b460591a56e9e31fa0ce8, entries=100, sequenceid=392, filesize=9.5 K 2023-04-24 08:59:17,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/.tmp/C/7e4aed69f18f46e49bf9ae7539d3730c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/7e4aed69f18f46e49bf9ae7539d3730c 2023-04-24 08:59:17,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/7e4aed69f18f46e49bf9ae7539d3730c, entries=100, sequenceid=392, filesize=9.5 K 2023-04-24 08:59:17,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 415f076aa60cbee3cb48b662f1bba024 in 1701ms, sequenceid=392, compaction requested=false 2023-04-24 08:59:17,946 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cfb1a87d60c34ac0ab4c8b7b19ca40de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/389f93ff27b148b79ab76f4318db7ed4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/395f37034acd4abe9a9770832e0c23d2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/247b6c1f60e445769c9f49c3a04be36c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cb9b58fb65ba4fdfb4cb2a038a6f90be, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d25db1a8613c4d178636a2f14454b261, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/71091321b3624cd99fec70ad8478a951, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/bad0e709fb5e4db790c15cb5eb16d754, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/e183dd4063c946d9870109d65bd14677, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/dc89408762aa43179b982e15b65c9613, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d38d093fa7d04666bc52a3023645f622, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/eb15d316bf314034b2f6d692f63e29c8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/2981b8096c4043359f5a7e64b1ab3b94, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fd2d0311be40423d8ea93439e65bf942, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f67950942941402caab8deb5b29b950e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c504acddaa804248bf6151dcdf21d5bf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f44d1ba1b3fb4f97869f709553784f9b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fecb8ed56cbb41bcb6da78c38e7ca4df, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c414b54c719148c2a825f779d409be95, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/583851d1df4b42dbbee3517a8d0c8fb9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/ea15131db3a746049c3c3e814fce4194, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/5d2739ce6f3743cfa742ab68773337ac, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/b62fdd6c9c0149d89ea58d0046e13fdc] to archive 2023-04-24 08:59:17,947 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:59:17,949 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cfb1a87d60c34ac0ab4c8b7b19ca40de to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cfb1a87d60c34ac0ab4c8b7b19ca40de 2023-04-24 08:59:17,950 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/389f93ff27b148b79ab76f4318db7ed4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/389f93ff27b148b79ab76f4318db7ed4 2023-04-24 08:59:17,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/395f37034acd4abe9a9770832e0c23d2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/395f37034acd4abe9a9770832e0c23d2 2023-04-24 08:59:17,951 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/247b6c1f60e445769c9f49c3a04be36c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/247b6c1f60e445769c9f49c3a04be36c 2023-04-24 08:59:17,952 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cb9b58fb65ba4fdfb4cb2a038a6f90be to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cb9b58fb65ba4fdfb4cb2a038a6f90be 2023-04-24 08:59:17,953 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d25db1a8613c4d178636a2f14454b261 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d25db1a8613c4d178636a2f14454b261 2023-04-24 08:59:17,954 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/71091321b3624cd99fec70ad8478a951 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/71091321b3624cd99fec70ad8478a951 2023-04-24 08:59:17,955 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/bad0e709fb5e4db790c15cb5eb16d754 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/bad0e709fb5e4db790c15cb5eb16d754 2023-04-24 08:59:17,956 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/e183dd4063c946d9870109d65bd14677 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/e183dd4063c946d9870109d65bd14677 2023-04-24 08:59:17,957 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/dc89408762aa43179b982e15b65c9613 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/dc89408762aa43179b982e15b65c9613 2023-04-24 08:59:17,958 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d38d093fa7d04666bc52a3023645f622 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/d38d093fa7d04666bc52a3023645f622 2023-04-24 08:59:17,960 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/eb15d316bf314034b2f6d692f63e29c8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/eb15d316bf314034b2f6d692f63e29c8 2023-04-24 08:59:17,962 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/2981b8096c4043359f5a7e64b1ab3b94 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/2981b8096c4043359f5a7e64b1ab3b94 2023-04-24 08:59:17,963 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fd2d0311be40423d8ea93439e65bf942 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fd2d0311be40423d8ea93439e65bf942 2023-04-24 08:59:17,964 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f67950942941402caab8deb5b29b950e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f67950942941402caab8deb5b29b950e 2023-04-24 08:59:17,966 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c504acddaa804248bf6151dcdf21d5bf to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c504acddaa804248bf6151dcdf21d5bf 2023-04-24 08:59:17,967 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f44d1ba1b3fb4f97869f709553784f9b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/f44d1ba1b3fb4f97869f709553784f9b 2023-04-24 08:59:17,968 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fecb8ed56cbb41bcb6da78c38e7ca4df to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/fecb8ed56cbb41bcb6da78c38e7ca4df 2023-04-24 08:59:17,969 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c414b54c719148c2a825f779d409be95 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c414b54c719148c2a825f779d409be95 2023-04-24 08:59:17,970 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/583851d1df4b42dbbee3517a8d0c8fb9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/583851d1df4b42dbbee3517a8d0c8fb9 2023-04-24 08:59:17,971 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/ea15131db3a746049c3c3e814fce4194 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/ea15131db3a746049c3c3e814fce4194 2023-04-24 08:59:17,972 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/5d2739ce6f3743cfa742ab68773337ac to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/5d2739ce6f3743cfa742ab68773337ac 2023-04-24 08:59:17,973 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/b62fdd6c9c0149d89ea58d0046e13fdc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/b62fdd6c9c0149d89ea58d0046e13fdc 2023-04-24 08:59:17,976 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/067c91315e984683810b3b6041e50b7b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/96e5f79d37014b7996bf40698626b537, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/0f748a096c12424985e9f5714aaaaf7b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/fe2ade4869b140f4a34e8f97ec5e251a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/171e4228b53c419eaa3e0b2938514825, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/9b0717842662431db1e64e49cd3a3cca, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/eb57e38a77cc45adb23cfda15635309d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c0291c94954043ec88af748edb0c5789, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/a0267282fcfc43519a7965d9b0a7cf7d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/abe1cc5031e940a5bfef341adf56d9f3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/be71fff4444a4509a8db7656462c024b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/062641df850843cea6218ff5f33268c5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/7a86ee6dbe4d41ba99bb9498376e98ec, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/84df5cae824e47cd85ad2ce1845f935d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/32f8216eb46947d3b54b177059a19a49, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/37d285d7a9dc4bb8ae179e7b7c7fbbe7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/922dafc4e5c541c8be08ac242129e4d0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/ff3cc6c892534caf80016c5c11fa419c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/339390390c07421ebb928dac7d256122, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/e24e0839ab57486c8a14625a753bd683, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/519c9a3200f04c3a96e733f5a5fe70f7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/4ce397f44f7644ca943a3c89e53a887a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c4cf0fdde6d74e02b354416e7165044c] to archive 2023-04-24 08:59:17,977 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:59:17,978 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/067c91315e984683810b3b6041e50b7b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/067c91315e984683810b3b6041e50b7b 2023-04-24 08:59:17,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/96e5f79d37014b7996bf40698626b537 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/96e5f79d37014b7996bf40698626b537 2023-04-24 08:59:17,979 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/0f748a096c12424985e9f5714aaaaf7b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/0f748a096c12424985e9f5714aaaaf7b 2023-04-24 08:59:17,980 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/fe2ade4869b140f4a34e8f97ec5e251a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/fe2ade4869b140f4a34e8f97ec5e251a 2023-04-24 08:59:17,981 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/171e4228b53c419eaa3e0b2938514825 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/171e4228b53c419eaa3e0b2938514825 2023-04-24 08:59:17,982 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/9b0717842662431db1e64e49cd3a3cca to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/9b0717842662431db1e64e49cd3a3cca 2023-04-24 08:59:17,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/eb57e38a77cc45adb23cfda15635309d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/eb57e38a77cc45adb23cfda15635309d 2023-04-24 08:59:17,983 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c0291c94954043ec88af748edb0c5789 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c0291c94954043ec88af748edb0c5789 2023-04-24 08:59:17,984 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/a0267282fcfc43519a7965d9b0a7cf7d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/a0267282fcfc43519a7965d9b0a7cf7d 2023-04-24 08:59:17,985 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/abe1cc5031e940a5bfef341adf56d9f3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/abe1cc5031e940a5bfef341adf56d9f3 2023-04-24 08:59:17,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/be71fff4444a4509a8db7656462c024b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/be71fff4444a4509a8db7656462c024b 2023-04-24 08:59:17,986 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/062641df850843cea6218ff5f33268c5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/062641df850843cea6218ff5f33268c5 2023-04-24 08:59:17,987 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/7a86ee6dbe4d41ba99bb9498376e98ec to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/7a86ee6dbe4d41ba99bb9498376e98ec 2023-04-24 08:59:17,988 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/84df5cae824e47cd85ad2ce1845f935d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/84df5cae824e47cd85ad2ce1845f935d 2023-04-24 08:59:17,989 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/32f8216eb46947d3b54b177059a19a49 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/32f8216eb46947d3b54b177059a19a49 2023-04-24 08:59:17,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/37d285d7a9dc4bb8ae179e7b7c7fbbe7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/37d285d7a9dc4bb8ae179e7b7c7fbbe7 2023-04-24 08:59:17,990 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/922dafc4e5c541c8be08ac242129e4d0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/922dafc4e5c541c8be08ac242129e4d0 2023-04-24 08:59:17,991 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/ff3cc6c892534caf80016c5c11fa419c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/ff3cc6c892534caf80016c5c11fa419c 2023-04-24 08:59:17,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/339390390c07421ebb928dac7d256122 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/339390390c07421ebb928dac7d256122 2023-04-24 08:59:17,992 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/e24e0839ab57486c8a14625a753bd683 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/e24e0839ab57486c8a14625a753bd683 2023-04-24 08:59:17,993 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/519c9a3200f04c3a96e733f5a5fe70f7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/519c9a3200f04c3a96e733f5a5fe70f7 2023-04-24 08:59:17,994 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/4ce397f44f7644ca943a3c89e53a887a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/4ce397f44f7644ca943a3c89e53a887a 2023-04-24 08:59:17,995 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c4cf0fdde6d74e02b354416e7165044c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/c4cf0fdde6d74e02b354416e7165044c 2023-04-24 08:59:17,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/6d266c46bc804b4789f6b185f41a12f9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/73184a6c7ca444d297425d87eedfd935, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/711341c702d84b318d79e21d7fb9d520, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8d6cb18eed46429fba33f94322107065, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/0c9ed9c66c324dfb97e94f68bee59ae0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/15af92ad29ca44f39052b6d5922cf0d1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/94da91d9868a4eaea7deaf4760d4f7a8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a0f3c024073d4944be26b5f4cf95978a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/be8c0419fb0f40bea0c9981effe71472, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/431baf35819c4de1b4721f1bbd725af2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3a1b7c9c588f437c8ab65d02591d3817, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/eecaeb0e96ec4d8aaae784853794827c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a27b0b89de3f417887cae4961cc37ddf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/aa2c1c81a09b48acb830684d990bee23, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/877b9a2a538f4cb19f6da179f8dc4682, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/56d8ab724b7f4546a965b7c91c64d5dd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/184a4652671743efaebe8183e17ad936, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a1e06a68f54d49da94cbbba5e65dc44a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/e546fee51d314992847c782c0a46c2a7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/85c8eb9e6bed4ed28c55445488721fb7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3cb478c9a404440c90815401cbf3d9d7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8b882626a8e6481cab959b1b7d647b29, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/dc910eddbabe408996bf0b49bd0815c3] to archive 2023-04-24 08:59:17,997 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:59:17,998 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/6d266c46bc804b4789f6b185f41a12f9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/6d266c46bc804b4789f6b185f41a12f9 2023-04-24 08:59:18,000 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/73184a6c7ca444d297425d87eedfd935 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/73184a6c7ca444d297425d87eedfd935 2023-04-24 08:59:18,001 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/711341c702d84b318d79e21d7fb9d520 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/711341c702d84b318d79e21d7fb9d520 2023-04-24 08:59:18,002 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8d6cb18eed46429fba33f94322107065 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8d6cb18eed46429fba33f94322107065 2023-04-24 08:59:18,003 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/0c9ed9c66c324dfb97e94f68bee59ae0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/0c9ed9c66c324dfb97e94f68bee59ae0 2023-04-24 08:59:18,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/15af92ad29ca44f39052b6d5922cf0d1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/15af92ad29ca44f39052b6d5922cf0d1 2023-04-24 08:59:18,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/94da91d9868a4eaea7deaf4760d4f7a8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/94da91d9868a4eaea7deaf4760d4f7a8 2023-04-24 08:59:18,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a0f3c024073d4944be26b5f4cf95978a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a0f3c024073d4944be26b5f4cf95978a 2023-04-24 08:59:18,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/be8c0419fb0f40bea0c9981effe71472 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/be8c0419fb0f40bea0c9981effe71472 2023-04-24 08:59:18,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/431baf35819c4de1b4721f1bbd725af2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/431baf35819c4de1b4721f1bbd725af2 2023-04-24 08:59:18,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3a1b7c9c588f437c8ab65d02591d3817 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3a1b7c9c588f437c8ab65d02591d3817 2023-04-24 08:59:18,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/eecaeb0e96ec4d8aaae784853794827c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/eecaeb0e96ec4d8aaae784853794827c 2023-04-24 08:59:18,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a27b0b89de3f417887cae4961cc37ddf to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a27b0b89de3f417887cae4961cc37ddf 2023-04-24 08:59:18,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/aa2c1c81a09b48acb830684d990bee23 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/aa2c1c81a09b48acb830684d990bee23 2023-04-24 08:59:18,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/877b9a2a538f4cb19f6da179f8dc4682 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/877b9a2a538f4cb19f6da179f8dc4682 2023-04-24 08:59:18,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/56d8ab724b7f4546a965b7c91c64d5dd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/56d8ab724b7f4546a965b7c91c64d5dd 2023-04-24 08:59:18,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/184a4652671743efaebe8183e17ad936 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/184a4652671743efaebe8183e17ad936 2023-04-24 08:59:18,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a1e06a68f54d49da94cbbba5e65dc44a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/a1e06a68f54d49da94cbbba5e65dc44a 2023-04-24 08:59:18,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/e546fee51d314992847c782c0a46c2a7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/e546fee51d314992847c782c0a46c2a7 2023-04-24 08:59:18,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/85c8eb9e6bed4ed28c55445488721fb7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/85c8eb9e6bed4ed28c55445488721fb7 2023-04-24 08:59:18,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3cb478c9a404440c90815401cbf3d9d7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/3cb478c9a404440c90815401cbf3d9d7 2023-04-24 08:59:18,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8b882626a8e6481cab959b1b7d647b29 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/8b882626a8e6481cab959b1b7d647b29 2023-04-24 08:59:18,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/dc910eddbabe408996bf0b49bd0815c3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/dc910eddbabe408996bf0b49bd0815c3 2023-04-24 08:59:18,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/recovered.edits/395.seqid, newMaxSeqId=395, maxSeqId=4 2023-04-24 08:59:18,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024. 2023-04-24 08:59:18,024 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 415f076aa60cbee3cb48b662f1bba024: 2023-04-24 08:59:18,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:18,027 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=415f076aa60cbee3cb48b662f1bba024, regionState=CLOSED 2023-04-24 08:59:18,027 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326758026"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326758026"}]},"ts":"1682326758026"} 2023-04-24 08:59:18,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-24 08:59:18,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 415f076aa60cbee3cb48b662f1bba024, server=jenkins-hbase16.apache.org,42837,1682326693550 in 1.9610 sec 2023-04-24 08:59:18,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-24 08:59:18,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=415f076aa60cbee3cb48b662f1bba024, UNASSIGN in 1.9700 sec 2023-04-24 08:59:18,031 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326758031"}]},"ts":"1682326758031"} 2023-04-24 08:59:18,031 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 08:59:18,045 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 08:59:18,046 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0520 sec 2023-04-24 08:59:18,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-24 08:59:18,104 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-24 08:59:18,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-24 08:59:18,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:18,107 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:18,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-24 08:59:18,107 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:18,114 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:18,117 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/recovered.edits] 2023-04-24 08:59:18,121 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c6297f2b1fca4aa69b9fd53a21f20f4a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/c6297f2b1fca4aa69b9fd53a21f20f4a 2023-04-24 08:59:18,123 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cd8d71f5215a428f95e293028d3b6a84 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/A/cd8d71f5215a428f95e293028d3b6a84 2023-04-24 08:59:18,126 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/5c519b62699642d1bc5626fb48aca679 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/5c519b62699642d1bc5626fb48aca679 2023-04-24 08:59:18,127 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/6c477336465b460591a56e9e31fa0ce8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/B/6c477336465b460591a56e9e31fa0ce8 2023-04-24 08:59:18,131 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/7e4aed69f18f46e49bf9ae7539d3730c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/7e4aed69f18f46e49bf9ae7539d3730c 2023-04-24 08:59:18,134 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/faaf7e3786fc4d1795825367cd654a00 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/C/faaf7e3786fc4d1795825367cd654a00 2023-04-24 08:59:18,139 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/recovered.edits/395.seqid to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024/recovered.edits/395.seqid 2023-04-24 08:59:18,141 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/415f076aa60cbee3cb48b662f1bba024 2023-04-24 08:59:18,141 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 08:59:18,141 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 08:59:18,142 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-24 08:59:18,145 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424093ab23aab564093898c72f005366ea7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424093ab23aab564093898c72f005366ea7 2023-04-24 08:59:18,146 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424112cf40f618948cea5fbb8a256ed3474 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424112cf40f618948cea5fbb8a256ed3474 2023-04-24 08:59:18,147 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042419dfcd7fdc8f45f48c7f763457472398 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042419dfcd7fdc8f45f48c7f763457472398 2023-04-24 08:59:18,148 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242b963a76fc004f72a250264423d0a3f9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242b963a76fc004f72a250264423d0a3f9 2023-04-24 08:59:18,149 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424322f6362daa3468383480763d1f4910f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424322f6362daa3468383480763d1f4910f 2023-04-24 08:59:18,150 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243f56accbe6c24cb2b6af10905c8c9788 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243f56accbe6c24cb2b6af10905c8c9788 2023-04-24 08:59:18,151 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042442f9837723134d7e853a9ada60714c9a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042442f9837723134d7e853a9ada60714c9a 2023-04-24 08:59:18,153 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424462399f3c1064ef0850a25539bc94c62 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424462399f3c1064ef0850a25539bc94c62 2023-04-24 08:59:18,154 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042456cf97c68c83487180438d45a0d2f9c4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042456cf97c68c83487180438d45a0d2f9c4 2023-04-24 08:59:18,156 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042462990b03d7f6480aae2b8a3f61abccf7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042462990b03d7f6480aae2b8a3f61abccf7 2023-04-24 08:59:18,157 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246574c3fac38e40099d63b08fde17cacc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246574c3fac38e40099d63b08fde17cacc 2023-04-24 08:59:18,159 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246bd657920fe0444ba697710a519af4d9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246bd657920fe0444ba697710a519af4d9 2023-04-24 08:59:18,160 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424746bcd6cc9ee4c4fbab87c98adf4c6a1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424746bcd6cc9ee4c4fbab87c98adf4c6a1 2023-04-24 08:59:18,161 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424aefccf06ec8d424ba42e693d716cc048 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424aefccf06ec8d424ba42e693d716cc048 2023-04-24 08:59:18,162 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b1129015650f49cd8d01219cbb74abea to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b1129015650f49cd8d01219cbb74abea 2023-04-24 08:59:18,163 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b2f3459ca2304d73910f28b66e437ec2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b2f3459ca2304d73910f28b66e437ec2 2023-04-24 08:59:18,164 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bd93e92323c44ada849490e3eb300d87 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bd93e92323c44ada849490e3eb300d87 2023-04-24 08:59:18,165 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d698e93d6f624d1c95a762139e664df1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d698e93d6f624d1c95a762139e664df1 2023-04-24 08:59:18,166 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fbcff9ed067d43018129d9fa6dbb34fb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fbcff9ed067d43018129d9fa6dbb34fb 2023-04-24 08:59:18,166 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 08:59:18,168 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:18,169 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 08:59:18,171 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 08:59:18,171 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:18,171 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 08:59:18,172 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682326758171"}]},"ts":"9223372036854775807"} 2023-04-24 08:59:18,173 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 08:59:18,173 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 415f076aa60cbee3cb48b662f1bba024, NAME => 'TestAcidGuarantees,,1682326727393.415f076aa60cbee3cb48b662f1bba024.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 08:59:18,173 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 08:59:18,173 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682326758173"}]},"ts":"9223372036854775807"} 2023-04-24 08:59:18,174 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 08:59:18,179 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:18,179 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 74 msec 2023-04-24 08:59:18,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-24 08:59:18,209 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-24 08:59:18,227 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 232), OpenFileDescriptor=538 (was 560), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=352 (was 345) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 170), AvailableMemoryMB=8453 (was 8604) 2023-04-24 08:59:18,240 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=538, MaxFileDescriptor=60000, SystemLoadAverage=352, ProcessCount=168, AvailableMemoryMB=8453 2023-04-24 08:59:18,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 08:59:18,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 08:59:18,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:18,244 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 08:59:18,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-24 08:59:18,245 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 08:59:18,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 08:59:18,246 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:18,247 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091 empty. 2023-04-24 08:59:18,247 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:18,247 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 08:59:18,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 08:59:18,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 08:59:18,658 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 08:59:18,663 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9ccb4fdcc4ce1f47af3dea271de2f091, NAME => 'TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp 2023-04-24 08:59:18,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 08:59:19,072 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:59:19,073 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9ccb4fdcc4ce1f47af3dea271de2f091, disabling compactions & flushes 2023-04-24 08:59:19,073 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:19,073 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:19,073 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. after waiting 0 ms 2023-04-24 08:59:19,073 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:19,074 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:19,074 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:19,080 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 08:59:19,083 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326759083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326759083"}]},"ts":"1682326759083"} 2023-04-24 08:59:19,085 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 08:59:19,087 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 08:59:19,087 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326759087"}]},"ts":"1682326759087"} 2023-04-24 08:59:19,089 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 08:59:19,128 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ccb4fdcc4ce1f47af3dea271de2f091, ASSIGN}] 2023-04-24 08:59:19,132 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ccb4fdcc4ce1f47af3dea271de2f091, ASSIGN 2023-04-24 08:59:19,134 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ccb4fdcc4ce1f47af3dea271de2f091, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=false 2023-04-24 08:59:19,286 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=9ccb4fdcc4ce1f47af3dea271de2f091, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:19,287 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326759286"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326759286"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326759286"}]},"ts":"1682326759286"} 2023-04-24 08:59:19,290 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:59:19,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 08:59:19,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:19,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9ccb4fdcc4ce1f47af3dea271de2f091, NAME => 'TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:59:19,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:59:19,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,451 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,453 DEBUG [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A 2023-04-24 08:59:19,453 DEBUG [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A 2023-04-24 08:59:19,454 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:19,454 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ccb4fdcc4ce1f47af3dea271de2f091 columnFamilyName A 2023-04-24 08:59:19,454 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.HStore(310): Store=9ccb4fdcc4ce1f47af3dea271de2f091/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:19,455 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,456 DEBUG [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B 2023-04-24 08:59:19,456 DEBUG [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B 2023-04-24 08:59:19,456 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:19,457 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ccb4fdcc4ce1f47af3dea271de2f091 columnFamilyName B 2023-04-24 08:59:19,457 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.HStore(310): Store=9ccb4fdcc4ce1f47af3dea271de2f091/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:19,457 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,459 DEBUG [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C 2023-04-24 08:59:19,459 DEBUG [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C 2023-04-24 08:59:19,459 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:19,459 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ccb4fdcc4ce1f47af3dea271de2f091 columnFamilyName C 2023-04-24 08:59:19,460 INFO [StoreOpener-9ccb4fdcc4ce1f47af3dea271de2f091-1] regionserver.HStore(310): Store=9ccb4fdcc4ce1f47af3dea271de2f091/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:19,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:19,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 08:59:19,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:19,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 08:59:19,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 9ccb4fdcc4ce1f47af3dea271de2f091; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65610919, jitterRate=-0.022321119904518127}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 08:59:19,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:19,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., pid=30, masterSystemTime=1682326759444 2023-04-24 08:59:19,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:19,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:19,469 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=9ccb4fdcc4ce1f47af3dea271de2f091, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:19,469 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326759469"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326759469"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326759469"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326759469"}]},"ts":"1682326759469"} 2023-04-24 08:59:19,471 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-24 08:59:19,471 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 in 180 msec 2023-04-24 08:59:19,472 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-24 08:59:19,472 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ccb4fdcc4ce1f47af3dea271de2f091, ASSIGN in 343 msec 2023-04-24 08:59:19,473 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 08:59:19,473 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326759473"}]},"ts":"1682326759473"} 2023-04-24 08:59:19,474 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 08:59:19,504 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 08:59:19,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2620 sec 2023-04-24 08:59:20,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-24 08:59:20,352 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-24 08:59:20,353 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x73ef029c to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,362 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f210b07, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,364 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,365 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34602, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,366 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 08:59:20,368 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:35596, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 08:59:20,369 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x2135e03b to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,379 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a45fd63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,379 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x5f2b45f8 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,387 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5191421e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,387 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x24e1e878 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,395 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21e07d99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,396 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x7e9921a7 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,404 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5590a424, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,404 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x5ffcf146 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,412 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68df7707, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,413 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x07742ca6 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,428 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14001400, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,428 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x353e11a8 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,437 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13091ab5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,437 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x12f583c6 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,445 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57836361, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,446 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x6635f17a to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,454 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5860b5c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,454 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x38204078 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:20,462 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71d7d827, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:20,469 DEBUG [hconnection-0x6d0a5173-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,469 DEBUG [hconnection-0x4c4dfeb7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,469 DEBUG [hconnection-0x6bac5c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:20,471 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34632, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,471 DEBUG [hconnection-0x4504bd6d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,473 DEBUG [hconnection-0x7c202109-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,476 DEBUG [hconnection-0x8823965-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,476 DEBUG [hconnection-0x1c27aec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,478 DEBUG [hconnection-0x47652708-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,480 DEBUG [hconnection-0x12c8ab85-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,481 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34630, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,478 DEBUG [hconnection-0x28739efc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:20,481 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,481 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,481 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,488 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,489 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:20,490 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:20,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:20,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:20,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:20,491 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,491 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34672, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,495 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:20,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:20,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:20,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:20,513 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,513 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 08:59:20,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:20,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:20,513 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:20,513 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:20,513 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:20,514 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:20,514 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:20,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:20,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:20,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:20,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:20,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:20,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:20,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:20,514 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:20,514 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:20,514 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:20,515 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:20,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:20,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:20,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:20,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:20,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,520 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,520 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,520 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:20,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@709bf2a7[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:20,523 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:20,523 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,528 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,528 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,528 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:20,528 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:20,528 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:20,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/4e07db666a3a466892c6699f9a2f11f0 2023-04-24 08:59:20,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326820529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,536 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:20,536 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:20,536 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:20,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326820531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:20,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:20,538 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:20,538 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:20,538 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:20,538 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:20,538 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:20,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326820536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326820536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326820536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,545 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,545 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:20,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,545 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,547 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,553 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:20,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:20,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:20,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:20,554 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:20,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:20,554 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:20,554 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:20,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:20,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:20,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326820637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326820639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326820642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326820642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326820642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,654 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:20,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:20,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:20,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:20,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:20,657 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:20,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:20,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:20,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:20,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,661 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:20,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:20,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:20,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:20,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:20,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:20,668 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:20,668 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:20,668 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:20,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:20,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:20,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:20,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:20,672 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:20,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:20,673 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:20,673 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:20,673 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:20,674 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:20,678 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:20,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dac21c4[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:20,680 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:20,680 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,686 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,686 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:20,686 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:20,686 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:20,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:20,694 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,694 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,695 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,695 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:20,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:20,697 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:20,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:20,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:20,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:20,697 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:20,711 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,711 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,711 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,712 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,713 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:20,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:20,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:20,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:20,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:20,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:20,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:20,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,720 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,720 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:20,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:20,720 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:20,820 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:20,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:20,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:20,823 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:20,823 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:20,823 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:20,823 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:20,823 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,823 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,828 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,828 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:20,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:20,828 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,828 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:20,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:20,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:20,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:20,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:20,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:20,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:20,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:20,829 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:20,829 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:20,829 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:20,829 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:20,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:20,829 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:20,830 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,830 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:20,830 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:20,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326820840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326820841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326820843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326820844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:20,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326820844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,858 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,858 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:20,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b6d801e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:20,859 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:20,859 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,869 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:20,870 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,870 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:20,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:20,870 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:20,878 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:20,878 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:20,878 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:20,878 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:20,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:20,880 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:20,880 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:20,880 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:20,880 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:20,880 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:20,886 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,886 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:20,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,886 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:20,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:20,887 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,888 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:20,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:20,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:20,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:20,895 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:20,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:20,895 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:20,895 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:20,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:20,895 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:20,895 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:20,895 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:20,895 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:20,895 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/a6947aabedca41b89c50c3de5c95a3ad 2023-04-24 08:59:20,995 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:20,997 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/e00b3afcb9384ee09c83187a15b55598 2023-04-24 08:59:20,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:20,998 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:20,998 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:20,998 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:20,998 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:20,998 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:20,998 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/4e07db666a3a466892c6699f9a2f11f0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/4e07db666a3a466892c6699f9a2f11f0 2023-04-24 08:59:21,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,004 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,004 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:21,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:21,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:21,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:21,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:21,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:21,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:21,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:21,005 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,005 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:21,006 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:21,006 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,006 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:21,006 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:21,006 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,006 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:21,006 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:21,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/4e07db666a3a466892c6699f9a2f11f0, entries=150, sequenceid=14, filesize=11.6 K 2023-04-24 08:59:21,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/a6947aabedca41b89c50c3de5c95a3ad as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/a6947aabedca41b89c50c3de5c95a3ad 2023-04-24 08:59:21,011 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/a6947aabedca41b89c50c3de5c95a3ad, entries=150, sequenceid=14, filesize=11.6 K 2023-04-24 08:59:21,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,011 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/e00b3afcb9384ee09c83187a15b55598 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e00b3afcb9384ee09c83187a15b55598 2023-04-24 08:59:21,012 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,012 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:21,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1844de14[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:21,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:21,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e00b3afcb9384ee09c83187a15b55598, entries=150, sequenceid=14, filesize=11.6 K 2023-04-24 08:59:21,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 504ms, sequenceid=14, compaction requested=false 2023-04-24 08:59:21,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:21,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:21,020 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,020 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:21,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:21,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:21,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:21,028 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,028 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:21,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:21,030 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:21,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:21,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:21,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:21,030 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:21,036 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,036 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,036 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,036 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,037 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,044 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,044 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,044 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,044 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,045 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,045 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:21,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:21,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,045 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:21,045 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,045 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,045 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,045 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:21,045 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:21,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:21,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 08:59:21,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:21,144 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:21,144 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:21,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:21,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:21,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:21,145 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:21,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:21,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:21,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:21,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:21,148 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:21,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:21,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:21,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,149 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326821149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326821149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,153 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,153 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326821150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,153 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,153 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326821151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326821151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:21,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:21,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:21,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:21,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:21,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:21,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:21,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:21,156 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,156 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:21,156 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:21,156 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:21,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:21,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:21,156 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:21,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,161 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,161 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,161 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:21,162 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:21,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37650971[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:21,162 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,170 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,170 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:21,170 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:21,170 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:21,170 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:21,178 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:21,178 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,178 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,178 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:21,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:21,180 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:21,180 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:21,180 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:21,180 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:21,180 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:21,186 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,186 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,186 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,186 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,187 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:21,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:21,195 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:21,195 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:21,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,195 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:21,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326821252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326821252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326821254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326821254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326821254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,295 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:21,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:21,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:21,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:21,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:21,298 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:21,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:21,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:21,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,303 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,303 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,303 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,303 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:21,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:21,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:21,304 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,304 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:21,304 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:21,304 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:21,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:21,311 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,311 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,311 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,311 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:21,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b0bbc58[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:21,312 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:21,313 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:21,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:21,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,320 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:21,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:21,347 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:21,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:21,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:21,349 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:21,349 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:21,349 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:21,349 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:21,349 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:21,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,363 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,369 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,370 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:21,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:21,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,370 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:21,370 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,370 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:21,370 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,370 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,370 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326821455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326821455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326821456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326821457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326821457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,470 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:21,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:21,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:21,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:21,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:21,473 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:21,473 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,473 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:21,473 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:21,473 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,474 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,478 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,478 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,478 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,478 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:21,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:21,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:21,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:21,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:21,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:21,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:21,486 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:21,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:21,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cf1ae57[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:21,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:21,495 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,495 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:21,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:21,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:21,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:21,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:21,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:21,505 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:21,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:21,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:21,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:21,505 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:21,511 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,511 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,511 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,512 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:21,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:21,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:21,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,520 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:21,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,520 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:21,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/0c08942549414627aa9b5b2d018933e8 2023-04-24 08:59:21,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/d97b333667304f3fab8037b1a0db4ded 2023-04-24 08:59:21,620 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:21,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:21,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:21,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:21,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:21,624 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:21,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:21,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:21,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,628 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,628 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,628 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,628 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:21,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:21,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:21,629 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:21,629 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,629 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:21,629 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:21,629 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:21,629 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,630 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:21,630 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:21,630 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,630 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:21,630 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:21,630 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,630 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:21,630 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:21,636 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,636 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,636 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,637 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:21,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52fb79a3[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:21,638 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:21,638 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,644 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:21,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:21,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:21,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:21,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,653 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,653 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,653 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:21,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:21,655 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:21,655 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:21,655 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:21,655 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:21,655 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:21,661 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,661 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,661 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,663 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,669 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,669 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,669 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,670 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:21,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:21,670 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,670 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,670 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,670 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,670 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,670 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,670 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:21,670 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:21,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:21,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326821758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326821758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326821760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326821760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:21,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326821761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,770 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:21,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:21,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:21,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:21,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:21,773 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:21,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:21,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:21,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,778 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,778 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,778 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:21,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:21,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:21,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:21,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:21,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:21,779 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:21,779 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:21,779 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:21,779 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:21,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:21,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:21,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:21,786 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:21,788 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:21,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d4adc0c[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:21,788 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,794 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:21,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:21,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:21,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:21,803 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,803 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,803 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,803 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:21,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:21,805 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:21,805 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:21,805 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:21,805 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:21,805 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:21,831 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,831 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,831 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,831 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,833 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:21,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:21,845 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,845 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,845 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,845 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:21,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:21,845 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:21,845 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,845 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,845 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,945 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:21,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:21,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:21,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:21,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:21,948 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:21,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:21,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:21,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:21,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:21,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:21,954 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:21,954 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:21,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:21,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:21,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:21,955 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,955 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:21,955 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:21,955 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:21,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:21,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:21,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:21,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:21,961 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,961 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,961 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,961 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:21,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b855081[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:21,963 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:21,963 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,969 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,970 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,970 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:21,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:21,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:21,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:21,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:21,978 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:21,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:21,979 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:21,980 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:21,980 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:21,980 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:21,980 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:21,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/d7430ed1ebfd439e8148e74be6841d27 2023-04-24 08:59:21,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:21,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:21,987 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,988 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/0c08942549414627aa9b5b2d018933e8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0c08942549414627aa9b5b2d018933e8 2023-04-24 08:59:21,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0c08942549414627aa9b5b2d018933e8, entries=150, sequenceid=34, filesize=11.6 K 2023-04-24 08:59:21,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/d97b333667304f3fab8037b1a0db4ded as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d97b333667304f3fab8037b1a0db4ded 2023-04-24 08:59:21,994 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:21,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:21,995 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:21,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:21,995 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:21,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:21,995 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,995 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:21,995 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:21,995 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,995 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:21,995 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:21,995 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:21,995 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:21,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:21,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d97b333667304f3fab8037b1a0db4ded, entries=150, sequenceid=34, filesize=11.6 K 2023-04-24 08:59:21,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/d7430ed1ebfd439e8148e74be6841d27 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/d7430ed1ebfd439e8148e74be6841d27 2023-04-24 08:59:22,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/d7430ed1ebfd439e8148e74be6841d27, entries=150, sequenceid=34, filesize=11.6 K 2023-04-24 08:59:22,000 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 856ms, sequenceid=34, compaction requested=false 2023-04-24 08:59:22,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:22,095 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:22,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:22,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:22,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:22,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:22,098 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:22,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:22,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:22,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:22,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:22,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:22,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,103 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:22,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:22,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:22,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:22,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:22,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:22,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:22,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:22,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:22,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:22,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:22,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:22,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:22,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:22,104 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 08:59:22,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:22,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:22,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:22,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:22,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:22,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:22,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:22,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:22,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682326762275, max timestamp=1682326762279Num uniques 859; 2023-04-24 08:59:22,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:22,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682326762275, max timestamp=1682326762279Num uniques 859; 2023-04-24 08:59:22,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:22,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682326762275, max timestamp=1682326762279Num uniques 859; 2023-04-24 08:59:22,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:22,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326822292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326822295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326822297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326822297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326822298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326822398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326822399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326822402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326822403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326822403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,511 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/ce958721b479466c92bacb232d899735 2023-04-24 08:59:22,522 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/e8ffedaca6bf4defaad373cdeeeac4ab 2023-04-24 08:59:22,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326822601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326822601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326822607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326822607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326822608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326822903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326822904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326822910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326822910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:22,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326822911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,931 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/1a31a8dfaee443ebb07c1fdf53efeafd 2023-04-24 08:59:22,935 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/ce958721b479466c92bacb232d899735 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ce958721b479466c92bacb232d899735 2023-04-24 08:59:22,939 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ce958721b479466c92bacb232d899735, entries=150, sequenceid=46, filesize=11.6 K 2023-04-24 08:59:22,940 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/e8ffedaca6bf4defaad373cdeeeac4ab as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/e8ffedaca6bf4defaad373cdeeeac4ab 2023-04-24 08:59:22,943 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/e8ffedaca6bf4defaad373cdeeeac4ab, entries=150, sequenceid=46, filesize=11.6 K 2023-04-24 08:59:22,944 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/1a31a8dfaee443ebb07c1fdf53efeafd as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1a31a8dfaee443ebb07c1fdf53efeafd 2023-04-24 08:59:22,948 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1a31a8dfaee443ebb07c1fdf53efeafd, entries=150, sequenceid=46, filesize=11.6 K 2023-04-24 08:59:22,949 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=134.18 KB/137400 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 845ms, sequenceid=46, compaction requested=true 2023-04-24 08:59:22,949 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:22,949 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:22,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:22,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:22,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:22,949 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:22,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:22,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:22,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:22,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:22,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a67d8b0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:22,987 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:22,987 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:22,994 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:22,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:22,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:22,995 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:22,995 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:22,995 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:22,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:22,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:23,003 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:23,003 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:23,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,003 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:23,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:23,005 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:23,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:23,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:23,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:23,005 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:23,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:23,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,011 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:23,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:23,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:23,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:23,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:23,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:23,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:23,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:23,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:23,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:23,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:23,020 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:23,020 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,020 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:23,020 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:23,020 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:23,020 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:23,020 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,120 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:23,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:23,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:23,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:23,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:23,123 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:23,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:23,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:23,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:23,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:23,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:23,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:23,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:23,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:23,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:23,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:23,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:23,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:23,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:23,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:23,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:23,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:23,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:23,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:23,223 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-24 08:59:23,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:23,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:23,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:23,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:23,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:23,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:23,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:23,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:23,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326823415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326823416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326823418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326823419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326823419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326823522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326823522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326823523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,550 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 08:59:23,655 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/3751e998818e4ddeb364a0814bb87ac1 2023-04-24 08:59:23,664 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/d0c8c7dbbe544043a40e8f32d7da2a70 2023-04-24 08:59:23,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326823724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326823724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:23,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:23,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326823725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326824028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326824029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326824029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,075 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/c4d745402d35466ea2552203fc4e2cc8 2023-04-24 08:59:24,078 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/3751e998818e4ddeb364a0814bb87ac1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/3751e998818e4ddeb364a0814bb87ac1 2023-04-24 08:59:24,081 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/3751e998818e4ddeb364a0814bb87ac1, entries=150, sequenceid=82, filesize=11.6 K 2023-04-24 08:59:24,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/d0c8c7dbbe544043a40e8f32d7da2a70 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d0c8c7dbbe544043a40e8f32d7da2a70 2023-04-24 08:59:24,087 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d0c8c7dbbe544043a40e8f32d7da2a70, entries=150, sequenceid=82, filesize=11.6 K 2023-04-24 08:59:24,088 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/c4d745402d35466ea2552203fc4e2cc8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c4d745402d35466ea2552203fc4e2cc8 2023-04-24 08:59:24,093 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c4d745402d35466ea2552203fc4e2cc8, entries=150, sequenceid=82, filesize=11.6 K 2023-04-24 08:59:24,095 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 872ms, sequenceid=82, compaction requested=true 2023-04-24 08:59:24,095 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:24,095 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:24,095 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:24,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:24,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:24,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:24,128 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,128 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:24,128 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:24,128 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:24,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:24,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d89e3e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:24,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:24,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:24,136 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:24,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:24,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:24,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:24,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:24,137 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,137 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:24,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:24,144 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:24,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:24,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:24,148 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:24,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:24,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:24,148 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:24,148 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:24,153 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,153 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,153 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,154 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,158 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,193 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:24,193 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:24,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:24,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:24,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:24,194 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:24,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,194 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:24,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:24,194 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:24,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:24,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:24,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:24,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:24,294 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:24,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:24,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:24,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:24,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:24,297 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:24,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:24,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:24,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:24,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:24,303 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,303 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:24,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:24,303 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,303 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:24,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:24,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:24,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:24,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:24,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:24,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:24,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:24,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:24,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:24,304 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:24,304 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:24,304 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 08:59:24,305 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:24,305 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:24,305 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:24,305 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:24,305 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:24,305 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:24,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326764437, max timestamp=1682326764451Num uniques 859; 2023-04-24 08:59:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326764437, max timestamp=1682326764451Num uniques 859; 2023-04-24 08:59:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326764437, max timestamp=1682326764451Num uniques 859; 2023-04-24 08:59:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 08:59:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:24,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:24,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326824493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326824495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326824532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326824534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326824535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326824598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326824598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,713 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/e00be84ca8414ca0b1720f92efb12649 2023-04-24 08:59:24,725 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/25bf4d3343054b1d8644dd0d63e9fa28 2023-04-24 08:59:24,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326824800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:24,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:24,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326824802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326825105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326825106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,133 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/f3649f88bf674d9eb2722cd4ebc711f5 2023-04-24 08:59:25,139 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/e00be84ca8414ca0b1720f92efb12649 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e00be84ca8414ca0b1720f92efb12649 2023-04-24 08:59:25,142 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e00be84ca8414ca0b1720f92efb12649, entries=100, sequenceid=93, filesize=9.3 K 2023-04-24 08:59:25,143 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/25bf4d3343054b1d8644dd0d63e9fa28 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/25bf4d3343054b1d8644dd0d63e9fa28 2023-04-24 08:59:25,146 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/25bf4d3343054b1d8644dd0d63e9fa28, entries=100, sequenceid=93, filesize=9.3 K 2023-04-24 08:59:25,147 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/f3649f88bf674d9eb2722cd4ebc711f5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/f3649f88bf674d9eb2722cd4ebc711f5 2023-04-24 08:59:25,150 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/f3649f88bf674d9eb2722cd4ebc711f5, entries=100, sequenceid=93, filesize=9.3 K 2023-04-24 08:59:25,151 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 847ms, sequenceid=93, compaction requested=true 2023-04-24 08:59:25,151 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:25,151 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:25,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:25,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:25,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:25,151 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:25,178 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,178 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:25,178 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:25,178 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:25,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:25,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e988f5[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:25,179 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:25,179 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:25,186 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:25,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:25,186 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:25,186 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:25,186 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,186 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:25,186 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:25,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,194 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:25,194 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:25,195 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:25,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:25,196 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:25,196 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:25,196 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:25,196 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:25,196 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:25,203 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,203 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,203 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:25,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:25,203 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,204 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:25,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:25,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:25,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:25,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:25,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:25,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:25,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:25,211 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:25,211 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:25,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:25,212 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:25,212 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:25,212 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,312 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:25,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:25,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:25,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:25,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:25,335 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:25,335 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:25,336 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:25,336 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:25,336 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,336 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:25,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:25,344 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:25,345 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,345 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:25,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:25,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:25,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:25,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:25,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:25,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:25,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:25,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:25,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:25,353 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:25,354 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:25,354 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-24 08:59:25,354 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:25,354 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:25,354 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:25,354 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:25,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:25,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:25,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:25,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:25,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326825551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326825552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326825555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326825609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326825610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326825656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326825656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326825660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,781 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/407a71be480240488272e9982a51713a 2023-04-24 08:59:25,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326825859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326825860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:25,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:25,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326825864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326826163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326826164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326826169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,189 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/869462c969e74c3e9d8c195ae60466ac 2023-04-24 08:59:26,198 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/020a08d7562747f2ad26e78f2ef36875 2023-04-24 08:59:26,202 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/407a71be480240488272e9982a51713a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/407a71be480240488272e9982a51713a 2023-04-24 08:59:26,207 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/407a71be480240488272e9982a51713a, entries=150, sequenceid=128, filesize=11.6 K 2023-04-24 08:59:26,208 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/869462c969e74c3e9d8c195ae60466ac as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/869462c969e74c3e9d8c195ae60466ac 2023-04-24 08:59:26,211 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/869462c969e74c3e9d8c195ae60466ac, entries=150, sequenceid=128, filesize=11.6 K 2023-04-24 08:59:26,212 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/020a08d7562747f2ad26e78f2ef36875 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/020a08d7562747f2ad26e78f2ef36875 2023-04-24 08:59:26,215 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/020a08d7562747f2ad26e78f2ef36875, entries=150, sequenceid=128, filesize=11.6 K 2023-04-24 08:59:26,215 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 861ms, sequenceid=128, compaction requested=true 2023-04-24 08:59:26,215 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:26,216 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:26,216 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:26,216 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:26,216 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,216 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:26,216 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:26,244 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,244 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:26,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:26,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:26,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:26,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5929c231[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:26,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:26,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:26,253 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:26,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:26,253 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:26,253 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:26,253 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:26,253 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,253 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:26,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,261 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:26,261 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:26,261 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:26,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:26,263 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:26,263 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:26,263 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:26,263 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:26,263 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:26,269 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,269 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:26,269 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:26,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,270 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:26,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:26,271 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:26,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:26,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:26,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:26,278 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:26,278 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:26,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,278 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:26,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:26,378 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:26,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:26,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:26,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:26,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:26,396 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:26,396 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:26,396 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:26,396 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:26,397 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,397 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:26,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:26,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:26,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:26,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:26,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:26,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:26,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:26,412 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:26,412 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:26,412 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:26,423 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:26,423 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:26,423 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:26,424 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:26,424 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 08:59:26,424 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:26,424 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:26,424 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:26,425 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:26,425 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:26,425 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:26,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:26,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:26,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326766635, max timestamp=1682326766666Num uniques 859; 2023-04-24 08:59:26,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:26,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326766635, max timestamp=1682326766666Num uniques 859; 2023-04-24 08:59:26,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:26,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326766635, max timestamp=1682326766666Num uniques 859; 2023-04-24 08:59:26,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:26,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326826684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326826685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326826686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326826686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326826687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326826790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326826793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326826793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326826793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326826793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,906 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/b6918d79eea447da9f0d57310a1b48f6 2023-04-24 08:59:26,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326826994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326826996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326826997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:26,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326826997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:26,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326826998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326827298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326827299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326827300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326827302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326827302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,315 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/28ac2f9419744260aa85e9dc8b6a3a60 2023-04-24 08:59:27,330 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/33eb6a2bd8254064a782ee915065c979 2023-04-24 08:59:27,333 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/b6918d79eea447da9f0d57310a1b48f6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/b6918d79eea447da9f0d57310a1b48f6 2023-04-24 08:59:27,337 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/b6918d79eea447da9f0d57310a1b48f6, entries=150, sequenceid=139, filesize=11.8 K 2023-04-24 08:59:27,338 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/28ac2f9419744260aa85e9dc8b6a3a60 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/28ac2f9419744260aa85e9dc8b6a3a60 2023-04-24 08:59:27,341 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/28ac2f9419744260aa85e9dc8b6a3a60, entries=150, sequenceid=139, filesize=11.8 K 2023-04-24 08:59:27,342 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/33eb6a2bd8254064a782ee915065c979 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/33eb6a2bd8254064a782ee915065c979 2023-04-24 08:59:27,346 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/33eb6a2bd8254064a782ee915065c979, entries=150, sequenceid=139, filesize=11.8 K 2023-04-24 08:59:27,346 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 922ms, sequenceid=139, compaction requested=true 2023-04-24 08:59:27,347 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:27,347 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:27,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:27,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:27,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:27,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:27,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:27,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:27,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:27,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:27,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d3f33fc[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:27,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:27,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:27,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:27,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:27,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:27,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:27,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:27,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:27,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:27,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:27,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:27,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:27,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:27,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:27,371 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:27,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:27,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:27,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:27,371 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:27,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:27,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:27,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:27,378 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:27,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:27,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:27,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:27,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:27,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:27,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:27,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:27,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:27,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:27,403 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:27,403 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:27,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:27,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,404 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:27,404 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:27,404 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:27,504 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:27,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:27,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:27,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:27,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:27,507 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:27,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:27,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:27,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:27,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,507 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:27,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:27,519 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:27,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,520 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:27,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:27,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:27,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:27,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:27,521 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:27,521 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:27,521 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:27,521 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:27,521 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:27,521 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:27,521 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:27,521 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-24 08:59:27,521 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:27,521 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:27,521 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:27,521 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:27,521 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:27,522 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:27,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:27,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326827812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326827815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326827816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326827820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326827821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326827922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,929 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/e364409c6fc24917b7eb844aaa56c15b 2023-04-24 08:59:27,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326827929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326827929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326827929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:27,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:27,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326827929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326828127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326828134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326828134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326828136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326828136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,359 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/701149c336104e1595fec32e32e98e4d 2023-04-24 08:59:28,368 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/cba0822c07574d7294aaac685f676a16 2023-04-24 08:59:28,372 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/e364409c6fc24917b7eb844aaa56c15b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e364409c6fc24917b7eb844aaa56c15b 2023-04-24 08:59:28,378 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e364409c6fc24917b7eb844aaa56c15b, entries=150, sequenceid=174, filesize=11.8 K 2023-04-24 08:59:28,378 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/701149c336104e1595fec32e32e98e4d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/701149c336104e1595fec32e32e98e4d 2023-04-24 08:59:28,382 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/701149c336104e1595fec32e32e98e4d, entries=150, sequenceid=174, filesize=11.8 K 2023-04-24 08:59:28,383 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/cba0822c07574d7294aaac685f676a16 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/cba0822c07574d7294aaac685f676a16 2023-04-24 08:59:28,389 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/cba0822c07574d7294aaac685f676a16, entries=150, sequenceid=174, filesize=11.8 K 2023-04-24 08:59:28,390 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 869ms, sequenceid=174, compaction requested=true 2023-04-24 08:59:28,390 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:28,390 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:28,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:28,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:28,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:28,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,403 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,403 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,403 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:28,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad1690b[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:28,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:28,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,411 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,411 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:28,411 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:28,411 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:28,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,412 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:28,419 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,419 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,419 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:28,420 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:28,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:28,422 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:28,422 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:28,422 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:28,422 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:28,422 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:28,428 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,428 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,428 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:28,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:28,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:28,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:28,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:28,437 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:28,437 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:28,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:28,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:28,438 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 08:59:28,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:28,438 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:28,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:28,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:28,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:28,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:28,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326828459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326828460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326828461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326828461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326828462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/a6d44802fdc34620b398f02f6879e555 2023-04-24 08:59:28,537 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:28,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:28,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:28,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:28,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:28,539 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:28,540 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:28,540 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:28,540 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:28,540 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,540 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,544 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:28,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:28,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:28,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:28,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:28,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:28,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:28,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:28,546 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:28,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,553 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,553 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,553 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:28,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eabe284[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:28,554 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:28,554 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,561 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,561 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:28,561 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:28,561 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:28,561 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,561 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:28,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326828565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326828565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326828566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326828566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,570 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:28,570 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:28,570 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:28,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326828567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:28,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:28,575 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:28,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:28,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:28,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:28,575 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:28,586 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,586 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:28,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,586 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,587 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,588 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,594 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:28,594 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:28,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:28,594 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:28,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:28,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:28,594 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:28,594 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,595 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:28,595 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:28,595 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,595 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:28,595 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:28,595 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,595 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,695 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:28,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:28,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:28,698 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:28,698 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:28,698 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:28,698 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:28,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,703 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:28,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,703 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:28,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:28,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:28,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:28,704 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:28,704 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:28,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:28,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:28,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:28,705 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,705 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:28,705 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:28,705 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:28,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:28,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:28,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:28,711 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,711 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,711 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,711 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:28,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69848f2a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:28,713 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:28,713 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,719 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:28,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:28,719 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:28,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:28,728 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,728 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,728 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:28,728 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:28,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:28,730 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:28,730 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:28,730 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:28,730 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:28,730 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:28,736 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,736 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,736 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:28,736 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,744 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:28,744 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:28,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:28,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:28,744 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:28,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:28,744 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:28,744 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,745 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,745 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:28,745 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,745 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:28,745 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,745 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,745 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:28,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326828770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326828771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326828771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326828772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:28,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326828774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,845 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:28,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:28,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:28,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:28,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:28,848 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:28,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:28,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:28,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:28,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:28,853 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:28,853 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,853 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:28,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:28,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:28,854 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:28,854 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:28,854 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:28,854 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:28,854 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:28,855 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,855 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:28,855 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:28,855 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,855 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:28,855 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:28,855 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,855 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:28,855 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:28,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,880 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/d445a287291c42149ffba3125b01515f 2023-04-24 08:59:28,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:28,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d30e135[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:28,882 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:28,882 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,894 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:28,895 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:28,895 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:28,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/258a58903cdf4fd9b50f9b7237de6a33 2023-04-24 08:59:28,895 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,895 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:28,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/a6d44802fdc34620b398f02f6879e555 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a6d44802fdc34620b398f02f6879e555 2023-04-24 08:59:28,903 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:28,903 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,903 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,903 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:28,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:28,905 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:28,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:28,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:28,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:28,905 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:28,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a6d44802fdc34620b398f02f6879e555, entries=200, sequenceid=186, filesize=14.1 K 2023-04-24 08:59:28,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/d445a287291c42149ffba3125b01515f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d445a287291c42149ffba3125b01515f 2023-04-24 08:59:28,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:28,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,918 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:28,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d445a287291c42149ffba3125b01515f, entries=150, sequenceid=186, filesize=11.8 K 2023-04-24 08:59:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/258a58903cdf4fd9b50f9b7237de6a33 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/258a58903cdf4fd9b50f9b7237de6a33 2023-04-24 08:59:28,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/258a58903cdf4fd9b50f9b7237de6a33, entries=150, sequenceid=186, filesize=11.8 K 2023-04-24 08:59:28,926 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 488ms, sequenceid=186, compaction requested=true 2023-04-24 08:59:28,926 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:28,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:28,927 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-24 08:59:28,927 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-24 08:59:28,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:28,927 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:28,928 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:28,928 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:28,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:28,928 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:28,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:28,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:28,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:28,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:28,928 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,928 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:28,928 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:28,928 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:28,928 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:28,929 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,929 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:28,929 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:28,933 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107470 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-24 08:59:28,933 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/A is initiating minor compaction (all files) 2023-04-24 08:59:28,933 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/A in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,933 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 105080 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-24 08:59:28,933 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/B is initiating minor compaction (all files) 2023-04-24 08:59:28,933 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/4e07db666a3a466892c6699f9a2f11f0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0c08942549414627aa9b5b2d018933e8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ce958721b479466c92bacb232d899735, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/3751e998818e4ddeb364a0814bb87ac1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e00be84ca8414ca0b1720f92efb12649, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/407a71be480240488272e9982a51713a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/b6918d79eea447da9f0d57310a1b48f6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e364409c6fc24917b7eb844aaa56c15b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a6d44802fdc34620b398f02f6879e555] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=105.0 K 2023-04-24 08:59:28,933 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/B in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,933 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/a6947aabedca41b89c50c3de5c95a3ad, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d97b333667304f3fab8037b1a0db4ded, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/e8ffedaca6bf4defaad373cdeeeac4ab, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d0c8c7dbbe544043a40e8f32d7da2a70, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/25bf4d3343054b1d8644dd0d63e9fa28, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/869462c969e74c3e9d8c195ae60466ac, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/28ac2f9419744260aa85e9dc8b6a3a60, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/701149c336104e1595fec32e32e98e4d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d445a287291c42149ffba3125b01515f] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=102.6 K 2023-04-24 08:59:28,933 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 4e07db666a3a466892c6699f9a2f11f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326760502 2023-04-24 08:59:28,933 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a6947aabedca41b89c50c3de5c95a3ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326760502 2023-04-24 08:59:28,934 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 0c08942549414627aa9b5b2d018933e8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682326760526 2023-04-24 08:59:28,934 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d97b333667304f3fab8037b1a0db4ded, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682326760526 2023-04-24 08:59:28,934 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting ce958721b479466c92bacb232d899735, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682326761149 2023-04-24 08:59:28,934 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e8ffedaca6bf4defaad373cdeeeac4ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682326761149 2023-04-24 08:59:28,934 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 3751e998818e4ddeb364a0814bb87ac1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326762296 2023-04-24 08:59:28,934 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d0c8c7dbbe544043a40e8f32d7da2a70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326762296 2023-04-24 08:59:28,935 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e00be84ca8414ca0b1720f92efb12649, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682326763418 2023-04-24 08:59:28,935 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 25bf4d3343054b1d8644dd0d63e9fa28, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682326763418 2023-04-24 08:59:28,935 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 407a71be480240488272e9982a51713a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326764469 2023-04-24 08:59:28,935 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting b6918d79eea447da9f0d57310a1b48f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1682326765550 2023-04-24 08:59:28,935 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e364409c6fc24917b7eb844aaa56c15b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1682326766678 2023-04-24 08:59:28,936 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 869462c969e74c3e9d8c195ae60466ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326764469 2023-04-24 08:59:28,940 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 28ac2f9419744260aa85e9dc8b6a3a60, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1682326765550 2023-04-24 08:59:28,940 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting a6d44802fdc34620b398f02f6879e555, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682326767814 2023-04-24 08:59:28,940 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 701149c336104e1595fec32e32e98e4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1682326766678 2023-04-24 08:59:28,941 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d445a287291c42149ffba3125b01515f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682326767814 2023-04-24 08:59:28,959 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#A#compaction#201 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:28,959 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#B#compaction#202 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:28,985 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/1205da37395c459e81e86161597fb2e9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1205da37395c459e81e86161597fb2e9 2023-04-24 08:59:28,989 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/bc7a6bf13b9347e893374b444f0b2afe as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/bc7a6bf13b9347e893374b444f0b2afe 2023-04-24 08:59:28,990 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/A of 9ccb4fdcc4ce1f47af3dea271de2f091 into 1205da37395c459e81e86161597fb2e9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:28,990 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:28,990 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/A, priority=7, startTime=1682326768927; duration=0sec 2023-04-24 08:59:28,991 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:28,991 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-24 08:59:28,995 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 105080 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-24 08:59:28,995 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/C is initiating minor compaction (all files) 2023-04-24 08:59:28,995 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/C in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:28,996 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e00b3afcb9384ee09c83187a15b55598, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/d7430ed1ebfd439e8148e74be6841d27, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1a31a8dfaee443ebb07c1fdf53efeafd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c4d745402d35466ea2552203fc4e2cc8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/f3649f88bf674d9eb2722cd4ebc711f5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/020a08d7562747f2ad26e78f2ef36875, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/33eb6a2bd8254064a782ee915065c979, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/cba0822c07574d7294aaac685f676a16, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/258a58903cdf4fd9b50f9b7237de6a33] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=102.6 K 2023-04-24 08:59:28,996 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e00b3afcb9384ee09c83187a15b55598, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326760502 2023-04-24 08:59:28,997 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting d7430ed1ebfd439e8148e74be6841d27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682326760526 2023-04-24 08:59:28,997 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1a31a8dfaee443ebb07c1fdf53efeafd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682326761149 2023-04-24 08:59:28,998 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c4d745402d35466ea2552203fc4e2cc8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326762296 2023-04-24 08:59:28,998 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f3649f88bf674d9eb2722cd4ebc711f5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1682326763418 2023-04-24 08:59:29,000 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 020a08d7562747f2ad26e78f2ef36875, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326764469 2023-04-24 08:59:29,000 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 33eb6a2bd8254064a782ee915065c979, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1682326765550 2023-04-24 08:59:29,001 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting cba0822c07574d7294aaac685f676a16, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1682326766678 2023-04-24 08:59:29,001 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/B of 9ccb4fdcc4ce1f47af3dea271de2f091 into bc7a6bf13b9347e893374b444f0b2afe(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:29,001 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:29,001 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/B, priority=7, startTime=1682326768927; duration=0sec 2023-04-24 08:59:29,001 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:29,001 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 258a58903cdf4fd9b50f9b7237de6a33, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682326767814 2023-04-24 08:59:29,028 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:29,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:29,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:29,032 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#C#compaction#203 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:29,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:29,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:29,072 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:29,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:29,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:29,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:29,076 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-24 08:59:29,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:29,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:29,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:29,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:29,077 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:29,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326768458, max timestamp=1682326769075Num uniques 859; 2023-04-24 08:59:29,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:29,077 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:29,078 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,078 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,078 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,078 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:29,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:29,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:29,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:29,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:29,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:29,090 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:29,090 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:29,091 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,091 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:29,091 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:29,091 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,091 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:29,091 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:29,091 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,091 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:29,091 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:29,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326829088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326829090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,095 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/e5593abaac0242aca2c2afeb2718652c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e5593abaac0242aca2c2afeb2718652c 2023-04-24 08:59:29,095 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,095 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,095 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,095 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:29,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0856a9[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:29,097 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:29,097 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/de5d66f6435f4f2f915923027f94a94d 2023-04-24 08:59:29,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326829092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326829093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326829093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,109 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/C of 9ccb4fdcc4ce1f47af3dea271de2f091 into e5593abaac0242aca2c2afeb2718652c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:29,109 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:29,109 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/C, priority=7, startTime=1682326768927; duration=0sec 2023-04-24 08:59:29,109 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:29,110 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,110 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,111 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:29,111 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:29,111 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:29,111 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:29,127 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:29,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:29,129 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:29,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:29,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:29,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:29,129 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:29,132 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:29,132 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,132 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:29,144 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,144 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,145 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,151 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,161 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:29,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:29,162 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,162 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,162 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,162 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,162 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,162 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:29,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326829195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326829195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326829203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326829203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326829203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,262 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:29,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:29,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:29,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:29,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:29,268 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:29,268 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,268 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:29,268 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:29,269 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,269 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,278 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,278 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,278 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:29,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:29,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:29,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:29,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:29,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:29,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:29,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:29,292 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,292 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:29,292 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:29,292 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:29,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:29,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:29,292 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:29,303 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,303 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,304 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:29,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d16206e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:29,305 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:29,305 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,311 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,311 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:29,311 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:29,311 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:29,311 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,311 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:29,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:29,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:29,325 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:29,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:29,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:29,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:29,325 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:29,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,337 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,338 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,345 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:29,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:29,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,345 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:29,345 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:29,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326829398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326829399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326829407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326829408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326829408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,445 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:29,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:29,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:29,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:29,448 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:29,448 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,448 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:29,448 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:29,449 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,449 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,453 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:29,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:29,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:29,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:29,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:29,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:29,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:29,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:29,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,461 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,461 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,461 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:29,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:29,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30261d39[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:29,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,489 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,489 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:29,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:29,489 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:29,489 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:29,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:29,502 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:29,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:29,505 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:29,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:29,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:29,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:29,505 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:29,511 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,511 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,511 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,512 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,516 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,528 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:29,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:29,528 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:29,528 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:29,530 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,530 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/f4f53b136ea4481993ea8a8d82b625e9 2023-04-24 08:59:29,629 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:29,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:29,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:29,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:29,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:29,633 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:29,633 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,633 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:29,633 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:29,633 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,634 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,644 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,644 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:29,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:29,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:29,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:29,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:29,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:29,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:29,647 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:29,647 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:29,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:29,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:29,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:29,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:29,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:29,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,653 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,653 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,653 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:29,654 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:29,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a656b41[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:29,654 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,661 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,661 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:29,661 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:29,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:29,661 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:29,669 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,670 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:29,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:29,671 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:29,672 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:29,672 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:29,672 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:29,672 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:29,678 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,678 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,678 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,678 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,679 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,686 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:29,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,687 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:29,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:29,687 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:29,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326829702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326829702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326829710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326829710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:29,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326829711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,787 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:29,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:29,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:29,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:29,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:29,790 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:29,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:29,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:29,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,794 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,794 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:29,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:29,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:29,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:29,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:29,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:29,796 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:29,796 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,796 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:29,796 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:29,797 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:29,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:29,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:29,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:29,803 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,803 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,803 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,803 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:29,804 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:29,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54c6c1ba[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:29,804 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:29,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:29,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:29,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:29,819 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:29,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:29,822 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:29,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:29,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:29,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:29,822 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:29,827 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,827 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,828 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,828 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,829 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:29,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:29,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:29,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:29,836 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:29,836 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:29,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:29,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,836 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:29,837 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,837 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,837 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,837 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,937 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:29,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:29,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:29,939 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:29,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:29,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:29,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,944 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:29,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:29,944 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,945 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:29,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:29,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:29,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:29,950 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,950 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:29,950 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:29,950 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:29,950 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:29,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,960 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:29,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:29,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:29,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@189f8e1c[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:29,962 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:29,962 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,969 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:29,969 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:29,969 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:29,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:29,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:29,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:29,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/1cef4573275b49578a9222bda8a46e28 2023-04-24 08:59:29,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/de5d66f6435f4f2f915923027f94a94d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/de5d66f6435f4f2f915923027f94a94d 2023-04-24 08:59:29,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/de5d66f6435f4f2f915923027f94a94d, entries=250, sequenceid=210, filesize=16.4 K 2023-04-24 08:59:29,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/f4f53b136ea4481993ea8a8d82b625e9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/f4f53b136ea4481993ea8a8d82b625e9 2023-04-24 08:59:29,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/f4f53b136ea4481993ea8a8d82b625e9, entries=150, sequenceid=210, filesize=11.8 K 2023-04-24 08:59:29,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/1cef4573275b49578a9222bda8a46e28 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1cef4573275b49578a9222bda8a46e28 2023-04-24 08:59:29,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1cef4573275b49578a9222bda8a46e28, entries=150, sequenceid=210, filesize=11.8 K 2023-04-24 08:59:29,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~96.16 KB/98470, heapSize ~281.98 KB/288752, currentSize=80.51 KB/82440 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 915ms, sequenceid=210, compaction requested=false 2023-04-24 08:59:29,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:30,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,002 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:30,003 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:30,003 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:30,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:30,005 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:30,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:30,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:30,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:30,005 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:30,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:30,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:30,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:30,011 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:30,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:30,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:30,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:30,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:30,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:30,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:30,020 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:30,020 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:30,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:30,120 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:30,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:30,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:30,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:30,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:30,123 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:30,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:30,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:30,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:30,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:30,128 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:30,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:30,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:30,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:30,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:30,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:30,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:30,129 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:30,129 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:30,129 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:30,129 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:30,129 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:30,129 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:30,129 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:30,129 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-24 08:59:30,129 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:30,130 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:30,130 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:30,130 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:30,130 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:30,130 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:30,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:30,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:30,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326830217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326830218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326830219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326830219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326830220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326830321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326830322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326830322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326830322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326830323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326830524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326830525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326830525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326830525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326830526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,536 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/339c45f18d4d473bad1e32684707acd1 2023-04-24 08:59:30,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326830826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326830829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326830829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326830829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:30,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34680 deadline: 1682326830831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,947 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/5a3e7fad4f3d46bca80bc971b3591e16 2023-04-24 08:59:30,958 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/bcad2a5cb5114187a164c0a73c646143 2023-04-24 08:59:30,961 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/339c45f18d4d473bad1e32684707acd1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/339c45f18d4d473bad1e32684707acd1 2023-04-24 08:59:30,964 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/339c45f18d4d473bad1e32684707acd1, entries=150, sequenceid=226, filesize=11.8 K 2023-04-24 08:59:30,965 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/5a3e7fad4f3d46bca80bc971b3591e16 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5a3e7fad4f3d46bca80bc971b3591e16 2023-04-24 08:59:30,968 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5a3e7fad4f3d46bca80bc971b3591e16, entries=150, sequenceid=226, filesize=11.8 K 2023-04-24 08:59:30,969 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/bcad2a5cb5114187a164c0a73c646143 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bcad2a5cb5114187a164c0a73c646143 2023-04-24 08:59:30,973 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bcad2a5cb5114187a164c0a73c646143, entries=150, sequenceid=226, filesize=11.8 K 2023-04-24 08:59:30,974 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 845ms, sequenceid=226, compaction requested=true 2023-04-24 08:59:30,974 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:30,974 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:30,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:30,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:30,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:30,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:30,978 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:30,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:30,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:30,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:30,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185097e3[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:30,980 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:30,980 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:30,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:30,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:30,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:30,986 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,986 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:30,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:30,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:30,994 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:30,994 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,994 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:30,994 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:30,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:30,996 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:30,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:30,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:30,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:30,996 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:31,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:31,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,004 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,011 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:31,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:31,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:31,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:31,011 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,011 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:31,111 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:31,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:31,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:31,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:31,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:31,114 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:31,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:31,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:31,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:31,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:31,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:31,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:31,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:31,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:31,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:31,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:31,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:31,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:31,121 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:31,121 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:31,121 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:31,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:31,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:31,121 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-24 08:59:31,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:31,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:31,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:31,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:31,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:31,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:31,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:31,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:31,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326831341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326831372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326831372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326831372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326831373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326831473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326831477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326831477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326831477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34670 deadline: 1682326831477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,528 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/75638e2eac734071badf5bd7a51a0e23 2023-04-24 08:59:31,539 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/5d985f97f44340c3a570a8684db8cd28 2023-04-24 08:59:31,550 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/73f6ad27e0304f12a1f496cc9bc91f61 2023-04-24 08:59:31,554 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/75638e2eac734071badf5bd7a51a0e23 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/75638e2eac734071badf5bd7a51a0e23 2023-04-24 08:59:31,559 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/75638e2eac734071badf5bd7a51a0e23, entries=150, sequenceid=243, filesize=11.8 K 2023-04-24 08:59:31,559 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/5d985f97f44340c3a570a8684db8cd28 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5d985f97f44340c3a570a8684db8cd28 2023-04-24 08:59:31,568 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5d985f97f44340c3a570a8684db8cd28, entries=150, sequenceid=243, filesize=11.8 K 2023-04-24 08:59:31,569 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/73f6ad27e0304f12a1f496cc9bc91f61 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/73f6ad27e0304f12a1f496cc9bc91f61 2023-04-24 08:59:31,573 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/73f6ad27e0304f12a1f496cc9bc91f61, entries=150, sequenceid=243, filesize=11.8 K 2023-04-24 08:59:31,574 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 453ms, sequenceid=243, compaction requested=true 2023-04-24 08:59:31,574 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:31,574 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:31,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:31,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:31,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:31,574 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:31,577 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,577 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,578 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,578 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:31,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790ec8cb[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:31,579 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:31,579 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,586 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:31,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:31,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:31,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:31,594 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:31,594 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,594 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,594 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:31,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:31,596 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:31,596 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:31,596 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:31,596 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:31,596 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:31,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,603 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:31,603 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,604 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:31,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:31,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:31,611 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:31,611 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,611 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:31,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:31,678 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-24 08:59:31,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:31,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:31,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:31,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:31,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:31,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:31,686 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/dc8ec1ed6c56478d83e6c10a4180c940 2023-04-24 08:59:31,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326831688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326831689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326831691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326831692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326831692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,711 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:31,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:31,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:31,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:31,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:31,714 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:31,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:31,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:31,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:31,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:31,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:31,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:31,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:31,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:31,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:31,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:31,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:31,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:31,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:31,720 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:31,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:31,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:31,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:31,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:31,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:31,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:31,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:31,727 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,727 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,728 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,728 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:31,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ba71d41[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:31,729 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:31,729 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,736 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:31,736 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,736 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:31,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:31,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:31,744 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:31,744 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,744 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,744 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:31,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:31,746 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:31,746 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:31,746 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:31,746 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:31,746 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:31,752 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,753 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:31,752 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:31,753 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,754 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,782 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,782 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:31,782 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:31,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:31,782 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:31,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:31,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:31,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:31,782 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,783 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:31,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:31,783 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:31,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,783 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:31,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:31,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326831793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326831793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326831795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326831796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326831796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,883 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:31,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:31,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:31,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:31,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:31,886 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:31,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:31,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:31,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:31,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:31,894 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:31,895 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,895 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:31,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:31,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:31,896 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:31,896 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:31,896 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:31,896 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:31,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:31,902 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,902 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,903 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,903 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:31,904 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:31,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fac9f47[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:31,904 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,911 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,911 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:31,911 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,911 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:31,911 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:31,911 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:31,919 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,919 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,919 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:31,919 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:31,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:31,921 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:31,921 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:31,921 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:31,921 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:31,921 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:31,927 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,927 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:31,928 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:31,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:31,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,929 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:31,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:31,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:31,936 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:31,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:31,936 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:31,936 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:31,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:31,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:31,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326831996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:31,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:31,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326831996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326831998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34630 deadline: 1682326831998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34616 deadline: 1682326831999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,037 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:32,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:32,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:32,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:32,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:32,039 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:32,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:32,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:32,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:32,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,044 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,044 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:32,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:32,044 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,044 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:32,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:32,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:32,046 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,046 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:32,046 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:32,046 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:32,046 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:32,052 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,053 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,053 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,053 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:32,054 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:32,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63dfe0de[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:32,054 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,061 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,061 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:32,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:32,061 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:32,061 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:32,069 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,069 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,069 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:32,069 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:32,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:32,071 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:32,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:32,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:32,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:32,071 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:32,077 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,077 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,078 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,078 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,079 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:32,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:32,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:32,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:32,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,086 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:32,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:32,086 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:32,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/1d5d3386dc5649f9b1f101c714654a47 2023-04-24 08:59:32,187 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:32,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:32,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:32,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:32,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:32,189 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:32,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:32,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:32,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:32,190 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,190 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,194 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,194 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:32,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:32,194 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,194 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:32,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:32,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:32,195 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:32,195 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:32,195 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:32,195 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:32,196 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:32,196 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,196 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:32,196 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:32,196 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,196 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:32,196 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:32,196 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,196 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:32,196 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:32,202 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,202 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,203 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,203 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:32,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9834fe[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:32,204 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:32,204 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,211 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:32,211 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:32,211 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,211 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:32,211 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:32,219 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:32,219 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,219 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,219 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:32,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:32,221 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:32,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:32,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:32,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:32,221 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:32,227 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,227 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,227 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,228 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,229 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,254 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,254 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:32,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:32,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:32,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:32,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:32,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:32,255 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:32,254 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:32,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,255 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:32,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,255 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:32,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:32,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:32,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326832300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326832300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326832302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326832302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326832303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,355 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:32,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:32,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:32,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:32,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:32,358 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:32,358 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:32,358 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:32,358 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:32,358 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,358 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,369 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:32,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:32,370 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,370 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:32,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:32,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:32,371 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,371 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:32,371 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:32,371 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:32,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:32,377 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,378 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:32,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:32,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ff66139[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:32,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,386 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:32,386 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:32,386 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:32,386 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,386 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:32,394 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:32,394 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,394 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,394 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:32,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:32,405 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:32,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:32,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:32,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:32,405 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:32,411 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,411 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:32,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:32,412 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:32,413 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:32,413 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:32,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:32,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:32,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:32,423 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:32,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:32,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:32,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:32,423 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:32,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:32,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:32,423 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:32,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/3d6011172cc34f7ba3bdf5a466642fd4 2023-04-24 08:59:32,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/dc8ec1ed6c56478d83e6c10a4180c940 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/dc8ec1ed6c56478d83e6c10a4180c940 2023-04-24 08:59:32,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/dc8ec1ed6c56478d83e6c10a4180c940, entries=200, sequenceid=260, filesize=14.2 K 2023-04-24 08:59:32,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/1d5d3386dc5649f9b1f101c714654a47 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1d5d3386dc5649f9b1f101c714654a47 2023-04-24 08:59:32,524 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:32,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:32,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:32,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:32,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:32,529 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:32,529 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:32,529 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:32,529 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:32,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1d5d3386dc5649f9b1f101c714654a47, entries=150, sequenceid=260, filesize=11.9 K 2023-04-24 08:59:32,530 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,530 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/3d6011172cc34f7ba3bdf5a466642fd4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/3d6011172cc34f7ba3bdf5a466642fd4 2023-04-24 08:59:32,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/3d6011172cc34f7ba3bdf5a466642fd4, entries=150, sequenceid=260, filesize=11.9 K 2023-04-24 08:59:32,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:32,536 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:32,536 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,536 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:32,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:32,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:32,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 861ms, sequenceid=260, compaction requested=true 2023-04-24 08:59:32,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:32,539 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:32,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:32,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:32,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:32,539 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:59:32,539 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:32,540 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:59:32,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:32,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:32,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:32,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:32,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:32,540 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-24 08:59:32,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:32,541 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:32,541 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:32,541 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:32,541 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:32,541 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:32,541 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 67808 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:59:32,541 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/A is initiating minor compaction (all files) 2023-04-24 08:59:32,541 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/A in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,541 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1205da37395c459e81e86161597fb2e9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/de5d66f6435f4f2f915923027f94a94d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/339c45f18d4d473bad1e32684707acd1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/75638e2eac734071badf5bd7a51a0e23, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/dc8ec1ed6c56478d83e6c10a4180c940] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=66.2 K 2023-04-24 08:59:32,544 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60588 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:59:32,544 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/B is initiating minor compaction (all files) 2023-04-24 08:59:32,544 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/B in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:32,544 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/bc7a6bf13b9347e893374b444f0b2afe, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/f4f53b136ea4481993ea8a8d82b625e9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5a3e7fad4f3d46bca80bc971b3591e16, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5d985f97f44340c3a570a8684db8cd28, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1d5d3386dc5649f9b1f101c714654a47] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=59.2 K 2023-04-24 08:59:32,544 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1205da37395c459e81e86161597fb2e9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682326767814 2023-04-24 08:59:32,544 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting de5d66f6435f4f2f915923027f94a94d, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682326768458 2023-04-24 08:59:32,544 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting bc7a6bf13b9347e893374b444f0b2afe, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682326767814 2023-04-24 08:59:32,545 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 339c45f18d4d473bad1e32684707acd1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682326769088 2023-04-24 08:59:32,545 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f4f53b136ea4481993ea8a8d82b625e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682326768458 2023-04-24 08:59:32,545 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5a3e7fad4f3d46bca80bc971b3591e16, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682326769088 2023-04-24 08:59:32,545 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 75638e2eac734071badf5bd7a51a0e23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682326770217 2023-04-24 08:59:32,546 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5d985f97f44340c3a570a8684db8cd28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682326770217 2023-04-24 08:59:32,546 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting dc8ec1ed6c56478d83e6c10a4180c940, keycount=200, bloomtype=ROW, size=14.2 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682326771340 2023-04-24 08:59:32,546 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 1d5d3386dc5649f9b1f101c714654a47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682326771340 2023-04-24 08:59:32,561 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#A#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:32,567 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#B#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:32,593 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/d69be73e0bee4054ab1601c521983bbd 2023-04-24 08:59:32,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:32,809 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:32,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326832819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326832819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326832820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326832821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34652 deadline: 1682326832821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326832924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326832924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326832924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326832927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:32,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326832927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:32,997 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/0204b54886364186b4fef08e20e8bf56 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0204b54886364186b4fef08e20e8bf56 2023-04-24 08:59:32,997 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/3f786becb11a4db7b41d22a425701c54 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3f786becb11a4db7b41d22a425701c54 2023-04-24 08:59:33,001 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/B of 9ccb4fdcc4ce1f47af3dea271de2f091 into 3f786becb11a4db7b41d22a425701c54(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:33,001 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/A of 9ccb4fdcc4ce1f47af3dea271de2f091 into 0204b54886364186b4fef08e20e8bf56(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:33,001 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:33,001 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/A, priority=11, startTime=1682326772539; duration=0sec 2023-04-24 08:59:33,001 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:33,001 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 08:59:33,001 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:33,002 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/B, priority=11, startTime=1682326772540; duration=0sec 2023-04-24 08:59:33,002 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:33,003 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60588 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 08:59:33,003 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/C is initiating minor compaction (all files) 2023-04-24 08:59:33,003 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/C in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,003 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e5593abaac0242aca2c2afeb2718652c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1cef4573275b49578a9222bda8a46e28, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bcad2a5cb5114187a164c0a73c646143, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/73f6ad27e0304f12a1f496cc9bc91f61, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/3d6011172cc34f7ba3bdf5a466642fd4] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=59.2 K 2023-04-24 08:59:33,003 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e5593abaac0242aca2c2afeb2718652c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1682326767814 2023-04-24 08:59:33,004 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1cef4573275b49578a9222bda8a46e28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682326768458 2023-04-24 08:59:33,004 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting bcad2a5cb5114187a164c0a73c646143, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1682326769088 2023-04-24 08:59:33,005 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 73f6ad27e0304f12a1f496cc9bc91f61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1682326770217 2023-04-24 08:59:33,005 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 3d6011172cc34f7ba3bdf5a466642fd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682326771340 2023-04-24 08:59:33,005 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/3cd274790a954dafb797ed8ce07330e7 2023-04-24 08:59:33,019 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#C#compaction#221 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:33,028 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/c5d2f6b7ce3e4af8892bbdfbc5c89764 2023-04-24 08:59:33,031 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/d69be73e0bee4054ab1601c521983bbd as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d69be73e0bee4054ab1601c521983bbd 2023-04-24 08:59:33,034 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d69be73e0bee4054ab1601c521983bbd, entries=150, sequenceid=276, filesize=11.9 K 2023-04-24 08:59:33,035 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/3cd274790a954dafb797ed8ce07330e7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3cd274790a954dafb797ed8ce07330e7 2023-04-24 08:59:33,043 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3cd274790a954dafb797ed8ce07330e7, entries=150, sequenceid=276, filesize=11.9 K 2023-04-24 08:59:33,044 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/c5d2f6b7ce3e4af8892bbdfbc5c89764 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c5d2f6b7ce3e4af8892bbdfbc5c89764 2023-04-24 08:59:33,048 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c5d2f6b7ce3e4af8892bbdfbc5c89764, entries=150, sequenceid=276, filesize=11.9 K 2023-04-24 08:59:33,049 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 509ms, sequenceid=276, compaction requested=false 2023-04-24 08:59:33,049 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:33,049 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,049 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:33,050 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:33,050 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,050 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:33,050 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:33,077 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,077 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,078 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,078 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:33,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@543b9927[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:33,079 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:33,079 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:33,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:33,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:33,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:33,094 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:33,094 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,094 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,094 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:33,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:33,096 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:33,096 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:33,096 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:33,096 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:33,096 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:33,102 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,102 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,102 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,104 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,111 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:33,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,111 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:33,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,112 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:33,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:33,129 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 08:59:33,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:33,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:33,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:33,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:33,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:33,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:33,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326833134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326833137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326833137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326833138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326833138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,212 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:33,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:33,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:33,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:33,214 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:33,214 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,214 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:33,214 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:33,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,219 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,219 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,220 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,220 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:33,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:33,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:33,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:33,220 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:33,220 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,220 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:33,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:33,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:33,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:33,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:33,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:33,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:33,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:33,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:33,227 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,227 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,228 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,228 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:33,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@487e4a6e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:33,229 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:33,229 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,236 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,236 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:33,236 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,236 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:33,236 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:33,236 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:33,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326833239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326833240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326833240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326833241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326833241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,244 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,244 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,244 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,244 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:33,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:33,246 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:33,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:33,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:33,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:33,246 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:33,252 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,252 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,253 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,253 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,254 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:33,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:33,261 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,261 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:33,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:33,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,261 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:33,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,362 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:33,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:33,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:33,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:33,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:33,364 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:33,364 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,364 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:33,364 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:33,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,412 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,412 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,412 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,412 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:33,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:33,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:33,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:33,413 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:33,413 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,413 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:33,413 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:33,413 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:33,413 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,413 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:33,414 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:33,414 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:33,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:33,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:33,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:33,427 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,427 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,428 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,428 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:33,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:33,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ecb85b[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:33,429 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,436 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:33,436 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:33,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:33,436 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:33,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326833441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326833443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326833443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326833443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326833444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:33,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:33,447 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:33,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:33,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:33,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:33,447 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:33,452 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,452 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,453 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,453 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/61cde1518c9f49dd84ec5dd504d24448 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/61cde1518c9f49dd84ec5dd504d24448 2023-04-24 08:59:33,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,455 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,457 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/C of 9ccb4fdcc4ce1f47af3dea271de2f091 into 61cde1518c9f49dd84ec5dd504d24448(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:33,457 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:33,457 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/C, priority=11, startTime=1682326772540; duration=0sec 2023-04-24 08:59:33,457 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:33,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:33,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,461 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,461 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/f30c68a4e8aa40f7b1d2f1076882aa01 2023-04-24 08:59:33,561 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:33,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:33,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:33,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:33,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:33,564 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:33,564 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,564 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:33,564 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:33,565 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,565 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,569 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,569 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,570 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,570 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:33,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:33,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:33,571 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,571 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:33,571 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:33,571 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:33,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:33,577 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,577 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,578 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,578 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:33,579 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:33,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19eff276[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:33,579 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,586 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,586 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:33,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:33,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:33,594 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:33,594 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,594 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,594 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:33,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:33,596 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:33,596 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:33,596 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:33,596 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:33,596 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:33,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,603 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,603 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,604 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:33,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:33,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,611 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:33,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,611 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:33,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,611 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,612 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,612 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,712 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:33,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:33,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:33,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:33,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:33,715 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:33,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:33,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:33,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:33,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:33,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:33,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:33,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:33,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:33,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:33,720 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:33,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:33,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:33,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:33,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:33,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:33,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,736 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,736 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:33,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1115a2cd[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:33,737 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:33,737 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326833746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326833746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326833747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326833748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:33,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326833749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,752 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,753 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:33,753 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,753 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:33,753 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:33,753 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:33,761 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:33,761 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,761 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,761 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:33,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:33,763 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:33,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:33,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:33,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:33,763 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:33,769 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,769 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,769 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,770 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,771 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,777 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,777 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,778 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:33,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:33,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:33,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,778 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,778 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:33,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,878 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:33,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:33,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:33,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:33,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:33,881 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:33,881 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,881 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:33,881 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:33,881 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,881 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,886 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:33,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:33,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:33,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:33,887 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:33,887 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:33,887 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:33,887 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:33,887 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:33,887 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,887 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:33,887 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:33,888 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:33,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:33,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:33,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:33,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:33,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,915 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:33,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:33,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b82b7bb[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:33,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,928 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,928 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:33,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,928 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:33,928 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:33,928 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:33,936 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:33,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,936 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,936 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:33,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:33,940 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:33,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:33,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:33,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:33,940 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:33,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:33,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:33,945 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:33,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/43faf627ac1d4aecaf32370295ebe53c 2023-04-24 08:59:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:33,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:33,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:33,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:33,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:33,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:33,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:33,953 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:33,953 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:33,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:34,053 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:34,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:34,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:34,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:34,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:34,057 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:34,057 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:34,057 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:34,057 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:34,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,061 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:34,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:34,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:34,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:34,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:34,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:34,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:34,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:34,062 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:34,062 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:34,062 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:34,062 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:34,062 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:34,062 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,062 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:34,062 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:34,063 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:34,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:34,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:34,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:34,069 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,069 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,070 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,070 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:34,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:34,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2cecf0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:34,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,078 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,078 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:34,078 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:34,078 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,078 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:34,078 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:34,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:34,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:34,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:34,091 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:34,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:34,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:34,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:34,091 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:34,102 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,102 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:34,102 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:34,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,104 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:34,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:34,112 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:34,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:34,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:34,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:34,112 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:34,112 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:34,212 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:34,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:34,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:34,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:34,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:34,215 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:34,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:34,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:34,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:34,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,219 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,219 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:34,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:34,219 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,219 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:34,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:34,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:34,220 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:34,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:34,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:34,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:34,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:34,227 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,227 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,228 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,228 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:34,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d36d7a1[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:34,229 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:34,229 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,236 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,236 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,236 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:34,236 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:34,236 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:34,236 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:34,244 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:34,244 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,244 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:34,244 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:34,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:34,246 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:34,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:34,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:34,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:34,246 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:34,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:34,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326834249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:34,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326834250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:34,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326834250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:34,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326834251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,252 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,252 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,253 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:34,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:34,253 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:34,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:34,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326834253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,254 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:34,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:34,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:34,261 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:34,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:34,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,261 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,261 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:34,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:34,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:34,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,262 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:34,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/4eeb0a0845894c6aa6be1f77ec7ceba8 2023-04-24 08:59:34,362 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:34,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:34,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:34,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:34,365 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:34,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:34,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:34,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:34,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/f30c68a4e8aa40f7b1d2f1076882aa01 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/f30c68a4e8aa40f7b1d2f1076882aa01 2023-04-24 08:59:34,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/f30c68a4e8aa40f7b1d2f1076882aa01, entries=200, sequenceid=298, filesize=14.3 K 2023-04-24 08:59:34,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/43faf627ac1d4aecaf32370295ebe53c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/43faf627ac1d4aecaf32370295ebe53c 2023-04-24 08:59:34,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:34,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:34,369 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:34,370 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,370 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:34,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:34,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:34,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:34,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:34,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:34,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:34,371 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,371 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:34,371 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:34,371 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:34,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:34,371 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:34,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:34,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/43faf627ac1d4aecaf32370295ebe53c, entries=150, sequenceid=298, filesize=11.9 K 2023-04-24 08:59:34,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/4eeb0a0845894c6aa6be1f77ec7ceba8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/4eeb0a0845894c6aa6be1f77ec7ceba8 2023-04-24 08:59:34,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/4eeb0a0845894c6aa6be1f77ec7ceba8, entries=150, sequenceid=298, filesize=11.9 K 2023-04-24 08:59:34,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 1255ms, sequenceid=298, compaction requested=true 2023-04-24 08:59:34,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:34,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:34,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:34,384 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:34,384 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:34,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:34,385 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:34,385 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/B is initiating minor compaction (all files) 2023-04-24 08:59:34,385 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/B in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,385 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3f786becb11a4db7b41d22a425701c54, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3cd274790a954dafb797ed8ce07330e7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/43faf627ac1d4aecaf32370295ebe53c] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=36.1 K 2023-04-24 08:59:34,385 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39426 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:34,385 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/A is initiating minor compaction (all files) 2023-04-24 08:59:34,385 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/A in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,385 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0204b54886364186b4fef08e20e8bf56, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d69be73e0bee4054ab1601c521983bbd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/f30c68a4e8aa40f7b1d2f1076882aa01] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=38.5 K 2023-04-24 08:59:34,386 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 0204b54886364186b4fef08e20e8bf56, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682326771340 2023-04-24 08:59:34,386 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,386 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,386 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:34,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,386 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3f786becb11a4db7b41d22a425701c54, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682326771340 2023-04-24 08:59:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,386 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting d69be73e0bee4054ab1601c521983bbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1682326771687 2023-04-24 08:59:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,387 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f30c68a4e8aa40f7b1d2f1076882aa01, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682326772820 2023-04-24 08:59:34,387 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3cd274790a954dafb797ed8ce07330e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1682326771687 2023-04-24 08:59:34,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,388 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 43faf627ac1d4aecaf32370295ebe53c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682326772820 2023-04-24 08:59:34,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:34,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71891700[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:34,389 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:34,389 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,394 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,395 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:34,395 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:34,395 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:34,395 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,395 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#A#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:34,395 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:34,395 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#B#compaction#226 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:34,423 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:34,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,424 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:34,424 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:34,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:34,426 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:34,426 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:34,426 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:34,426 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:34,426 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:34,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:34,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:34,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:34,437 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:34,444 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:34,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:34,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:34,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:34,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:34,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,445 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,445 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:34,445 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,445 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:34,445 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:34,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:34,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:34,545 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:34,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:34,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:34,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:34,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:34,547 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:34,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:34,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:34,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:34,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,553 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:34,553 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:34,553 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:34,553 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:34,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:34,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:34,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:34,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:34,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:34,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:34,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:34,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:34,554 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,554 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:34,554 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 08:59:34,554 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:34,555 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:34,555 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:34,555 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:34,555 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:34,555 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:34,804 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/0895948cca934beb903fc0deeab89b83 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/0895948cca934beb903fc0deeab89b83 2023-04-24 08:59:34,805 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/ad1f4c5dfd794139a05795dc942e80f5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ad1f4c5dfd794139a05795dc942e80f5 2023-04-24 08:59:34,809 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/B of 9ccb4fdcc4ce1f47af3dea271de2f091 into 0895948cca934beb903fc0deeab89b83(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:34,809 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/A of 9ccb4fdcc4ce1f47af3dea271de2f091 into ad1f4c5dfd794139a05795dc942e80f5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:34,809 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:34,809 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:34,809 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/B, priority=13, startTime=1682326774384; duration=0sec 2023-04-24 08:59:34,809 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/A, priority=13, startTime=1682326774384; duration=0sec 2023-04-24 08:59:34,809 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:34,809 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:34,809 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:34,810 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:34,810 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/C is initiating minor compaction (all files) 2023-04-24 08:59:34,810 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/C in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:34,811 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/61cde1518c9f49dd84ec5dd504d24448, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c5d2f6b7ce3e4af8892bbdfbc5c89764, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/4eeb0a0845894c6aa6be1f77ec7ceba8] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=36.1 K 2023-04-24 08:59:34,811 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 61cde1518c9f49dd84ec5dd504d24448, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1682326771340 2023-04-24 08:59:34,811 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c5d2f6b7ce3e4af8892bbdfbc5c89764, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1682326771687 2023-04-24 08:59:34,811 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4eeb0a0845894c6aa6be1f77ec7ceba8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682326772820 2023-04-24 08:59:34,816 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#C#compaction#228 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:34,962 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/21e4dc3e35de4d1ca6caabb4ef642ee8 2023-04-24 08:59:35,223 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/e055d1d33523416fa7d40a1e14c7c802 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e055d1d33523416fa7d40a1e14c7c802 2023-04-24 08:59:35,227 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/C of 9ccb4fdcc4ce1f47af3dea271de2f091 into e055d1d33523416fa7d40a1e14c7c802(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:35,227 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:35,227 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/C, priority=13, startTime=1682326774384; duration=0sec 2023-04-24 08:59:35,227 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:35,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:35,253 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:35,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326835260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326835261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326835262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326835262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326835263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326835364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326835364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326835365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326835365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326835365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,371 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/5415009cd6c64410a317be40e8f803aa 2023-04-24 08:59:35,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326835567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326835567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326835567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326835568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326835569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,797 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/b80a2a365d5440c2a6a89c0b215cd450 2023-04-24 08:59:35,800 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/21e4dc3e35de4d1ca6caabb4ef642ee8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/21e4dc3e35de4d1ca6caabb4ef642ee8 2023-04-24 08:59:35,804 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/21e4dc3e35de4d1ca6caabb4ef642ee8, entries=150, sequenceid=311, filesize=11.9 K 2023-04-24 08:59:35,804 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/5415009cd6c64410a317be40e8f803aa as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5415009cd6c64410a317be40e8f803aa 2023-04-24 08:59:35,807 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5415009cd6c64410a317be40e8f803aa, entries=150, sequenceid=311, filesize=11.9 K 2023-04-24 08:59:35,808 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/b80a2a365d5440c2a6a89c0b215cd450 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/b80a2a365d5440c2a6a89c0b215cd450 2023-04-24 08:59:35,811 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/b80a2a365d5440c2a6a89c0b215cd450, entries=150, sequenceid=311, filesize=11.9 K 2023-04-24 08:59:35,811 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 1257ms, sequenceid=311, compaction requested=false 2023-04-24 08:59:35,811 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:35,811 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:35,812 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:35,812 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:35,812 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,812 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:35,812 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:35,819 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:35,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:35,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:35,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f0859e5[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:35,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:35,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,827 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,828 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:35,828 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:35,828 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:35,828 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,828 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:35,836 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:35,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,836 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:35,836 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:35,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:35,838 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:35,838 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:35,838 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:35,838 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:35,838 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:35,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:35,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:35,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:35,845 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:35,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:35,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:35,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,845 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:35,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:35,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:35,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:35,853 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:35,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:35,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:35,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:35,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:35,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:35,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:35,853 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:35,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,853 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:35,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:35,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:35,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-24 08:59:35,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326775258, max timestamp=1682326775870Num uniques 859; 2023-04-24 08:59:35,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:35,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326775258, max timestamp=1682326775870Num uniques 859; 2023-04-24 08:59:35,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326775258, max timestamp=1682326775870Num uniques 859; 2023-04-24 08:59:35,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:35,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 08:59:35,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:35,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:35,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:35,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:35,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:35,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:35,879 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/d556a356edd441bf8556da5c7cf6902c 2023-04-24 08:59:35,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326775875, max timestamp=1682326775888Num uniques 859; 2023-04-24 08:59:35,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:35,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326775875, max timestamp=1682326775891Num uniques 859; 2023-04-24 08:59:35,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326775875, max timestamp=1682326775891Num uniques 859; 2023-04-24 08:59:35,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:35,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 08:59:35,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326775888, max timestamp=1682326775910Num uniques 1068; 2023-04-24 08:59:35,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 08:59:35,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326775888, max timestamp=1682326775910Num uniques 1068; 2023-04-24 08:59:35,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 08:59:35,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326775888, max timestamp=1682326775910Num uniques 1068; 2023-04-24 08:59:35,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-24 08:59:35,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326835949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326835949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:35,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326835949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8095, totalHeapSize=18648, min timestamp=1682326775942, max timestamp=1682326775951Num uniques 1127; 2023-04-24 08:59:35,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=51605, new segment data size=8095, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 08:59:35,953 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:35,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1682326775942, max timestamp=1682326775951Num uniques 1127; 2023-04-24 08:59:35,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682326775942, max timestamp=1682326775951Num uniques 1127; 2023-04-24 08:59:35,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:35,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 08:59:35,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=51602, new segment data size=8092, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 08:59:35,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:35,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:35,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:35,956 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:35,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:35,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:35,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:35,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:35,961 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:35,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:35,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:35,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:35,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:35,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:35,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:35,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:35,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:35,962 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:35,962 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:35,962 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:35,962 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:35,962 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:35,962 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:35,963 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:35,963 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:35,963 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:35,963 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:35,963 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:35,963 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,963 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:35,963 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:35,969 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:35,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:35,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:35,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30f7798[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:35,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:35,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,977 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:35,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:35,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:35,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:35,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:35,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326775962, max timestamp=1682326775981Num uniques 1036; 2023-04-24 08:59:35,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-24 08:59:35,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326775962, max timestamp=1682326775981Num uniques 1036; 2023-04-24 08:59:35,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326775962, max timestamp=1682326775981Num uniques 1036; 2023-04-24 08:59:35,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-24 08:59:35,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-24 08:59:35,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:35,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:35,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:35,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:35,988 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:35,988 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:35,988 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:35,988 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:35,988 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:35,994 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,994 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,994 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:35,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:35,995 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:36,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:36,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:36,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,003 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:36,003 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,003 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:36,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326776002, max timestamp=1682326776011Num uniques 1095; 2023-04-24 08:59:36,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326776002, max timestamp=1682326776011Num uniques 1095; 2023-04-24 08:59:36,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 08:59:36,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326776002, max timestamp=1682326776011Num uniques 1095; 2023-04-24 08:59:36,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 08:59:36,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 08:59:36,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682326776019, max timestamp=1682326776037Num uniques 1104; 2023-04-24 08:59:36,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 08:59:36,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326836037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682326776019, max timestamp=1682326776037Num uniques 1104; 2023-04-24 08:59:36,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682326776019, max timestamp=1682326776037Num uniques 1104; 2023-04-24 08:59:36,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 08:59:36,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 08:59:36,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682326776060, max timestamp=1682326776065Num uniques 1013; 2023-04-24 08:59:36,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-24 08:59:36,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682326776060, max timestamp=1682326776065Num uniques 1013; 2023-04-24 08:59:36,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682326776060, max timestamp=1682326776065Num uniques 1013; 2023-04-24 08:59:36,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-24 08:59:36,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-24 08:59:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682326776082, max timestamp=1682326776088Num uniques 1072; 2023-04-24 08:59:36,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-24 08:59:36,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682326776082, max timestamp=1682326776088Num uniques 1072; 2023-04-24 08:59:36,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-24 08:59:36,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682326776082, max timestamp=1682326776088Num uniques 1072; 2023-04-24 08:59:36,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-24 08:59:36,103 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:36,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:36,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:36,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:36,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:36,106 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:36,106 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,106 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:36,106 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:36,106 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,106 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,111 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,111 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,111 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,111 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:36,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:36,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:36,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:36,112 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:36,112 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,112 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:36,112 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:36,112 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:36,112 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,112 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:36,112 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:36,112 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,113 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:36,113 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:36,113 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,113 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:36,113 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:36,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682326776097, max timestamp=1682326776112Num uniques 1081; 2023-04-24 08:59:36,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-24 08:59:36,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682326776097, max timestamp=1682326776112Num uniques 1081; 2023-04-24 08:59:36,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-24 08:59:36,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1682326776097, max timestamp=1682326776112Num uniques 1081; 2023-04-24 08:59:36,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-24 08:59:36,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,119 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,119 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,119 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:36,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@787f2387[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:36,120 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:36,121 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,127 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,128 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:36,128 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:36,128 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:36,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,128 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:36,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326836128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682326776125, max timestamp=1682326776130Num uniques 1140; 2023-04-24 08:59:36,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-24 08:59:36,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682326776125, max timestamp=1682326776130Num uniques 1140; 2023-04-24 08:59:36,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=13280, suffix heap size=146528,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=290 2023-04-24 08:59:36,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682326776126, max timestamp=1682326776130Num uniques 1140; 2023-04-24 08:59:36,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-24 08:59:36,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:36,136 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:36,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:36,138 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:36,138 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:36,138 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:36,138 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:36,138 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:36,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682326776142, max timestamp=1682326776152Num uniques 1149; 2023-04-24 08:59:36,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-24 08:59:36,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682326776142, max timestamp=1682326776152Num uniques 1149; 2023-04-24 08:59:36,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-24 08:59:36,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682326776142, max timestamp=1682326776152Num uniques 1049; 2023-04-24 08:59:36,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-24 08:59:36,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,162 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,162 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326776170, max timestamp=1682326776177Num uniques 1108; 2023-04-24 08:59:36,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326776170, max timestamp=1682326776177Num uniques 1108; 2023-04-24 08:59:36,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 08:59:36,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 08:59:36,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326776170, max timestamp=1682326776177Num uniques 1108; 2023-04-24 08:59:36,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 08:59:36,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,195 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:36,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,195 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,196 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,196 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,196 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,196 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,196 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:36,196 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:36,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1682326776183, max timestamp=1682326776199Num uniques 1067; 2023-04-24 08:59:36,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1682326776183, max timestamp=1682326776199Num uniques 1067; 2023-04-24 08:59:36,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-24 08:59:36,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-24 08:59:36,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1682326776183, max timestamp=1682326776199Num uniques 1067; 2023-04-24 08:59:36,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-24 08:59:36,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326836219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1682326776212, max timestamp=1682326776220Num uniques 1126; 2023-04-24 08:59:36,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-04-24 08:59:36,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1682326776212, max timestamp=1682326776220Num uniques 1126; 2023-04-24 08:59:36,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-04-24 08:59:36,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1682326776212, max timestamp=1682326776220Num uniques 1126; 2023-04-24 08:59:36,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-04-24 08:59:36,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326836282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326836282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326836282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682326776256, max timestamp=1682326776283Num uniques 1135; 2023-04-24 08:59:36,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=8470, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-24 08:59:36,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682326776256, max timestamp=1682326776283Num uniques 1135; 2023-04-24 08:59:36,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=8470, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-24 08:59:36,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682326776256, max timestamp=1682326776283Num uniques 1135; 2023-04-24 08:59:36,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=8470, suffix heap size=146008,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=185 2023-04-24 08:59:36,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/57c1ceb052a541139f3711756c2e19b8 2023-04-24 08:59:36,296 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:36,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:36,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:36,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:36,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:36,300 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:36,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:36,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:36,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,300 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,311 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,311 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,311 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,311 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:36,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:36,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:36,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:36,312 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:36,312 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,312 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:36,312 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:36,312 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:36,312 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:36,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:36,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:36,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:36,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:36,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:36,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:36,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f71a9f[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:36,321 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:36,321 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,327 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,328 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,328 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:36,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:36,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:36,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:36,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682326776311, max timestamp=1682326776331Num uniques 1044; 2023-04-24 08:59:36,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-24 08:59:36,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682326776311, max timestamp=1682326776331Num uniques 1044; 2023-04-24 08:59:36,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-24 08:59:36,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682326776311, max timestamp=1682326776331Num uniques 1044; 2023-04-24 08:59:36,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-24 08:59:36,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:36,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:36,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:36,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:36,338 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:36,338 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:36,338 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:36,338 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:36,338 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:36,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,345 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,346 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:36,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,353 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,353 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:36,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,353 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1682326776339, max timestamp=1682326776370Num uniques 1103; 2023-04-24 08:59:36,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=50508, new segment data size=9288, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-24 08:59:36,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682326776339, max timestamp=1682326776370Num uniques 1103; 2023-04-24 08:59:36,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-24 08:59:36,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682326776339, max timestamp=1682326776370Num uniques 1103; 2023-04-24 08:59:36,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-24 08:59:36,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326836391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326836392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326836393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682326776390, max timestamp=1682326776393Num uniques 1062; 2023-04-24 08:59:36,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-24 08:59:36,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682326776390, max timestamp=1682326776393Num uniques 1062; 2023-04-24 08:59:36,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682326776390, max timestamp=1682326776393Num uniques 1062; 2023-04-24 08:59:36,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-24 08:59:36,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-24 08:59:36,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682326776418, max timestamp=1682326776427Num uniques 1021; 2023-04-24 08:59:36,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-24 08:59:36,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682326776418, max timestamp=1682326776427Num uniques 1021; 2023-04-24 08:59:36,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682326776418, max timestamp=1682326776427Num uniques 1021; 2023-04-24 08:59:36,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-24 08:59:36,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-24 08:59:36,453 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:36,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:36,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:36,456 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:36,456 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,456 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:36,456 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:36,456 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,456 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:36,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:36,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:36,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:36,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:36,462 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:36,462 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,462 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:36,462 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:36,462 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:36,462 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,462 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:36,462 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:36,463 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:36,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682326776455, max timestamp=1682326776464Num uniques 1030; 2023-04-24 08:59:36,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-24 08:59:36,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682326776455, max timestamp=1682326776464Num uniques 1030; 2023-04-24 08:59:36,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-24 08:59:36,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682326776455, max timestamp=1682326776464Num uniques 1030; 2023-04-24 08:59:36,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-24 08:59:36,469 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,469 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,469 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,469 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:36,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3da435[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:36,471 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:36,471 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,477 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,478 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:36,478 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,478 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:36,478 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:36,478 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:36,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:36,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,486 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:36,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:36,488 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:36,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:36,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:36,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:36,488 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:36,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682326776484, max timestamp=1682326776489Num uniques 1039; 2023-04-24 08:59:36,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-24 08:59:36,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682326776484, max timestamp=1682326776489Num uniques 1039; 2023-04-24 08:59:36,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-24 08:59:36,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1682326776484, max timestamp=1682326776489Num uniques 1039; 2023-04-24 08:59:36,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-24 08:59:36,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,495 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,496 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,502 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,502 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:36,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:36,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,503 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,503 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:36,503 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:36,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1682326776505, max timestamp=1682326776508Num uniques 1048; 2023-04-24 08:59:36,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=9068, suffix heap size=136960,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=198 2023-04-24 08:59:36,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1682326776505, max timestamp=1682326776508Num uniques 1048; 2023-04-24 08:59:36,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=9068, suffix heap size=136960,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=198 2023-04-24 08:59:36,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1682326776505, max timestamp=1682326776508Num uniques 1048; 2023-04-24 08:59:36,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=9068, suffix heap size=136960,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=198 2023-04-24 08:59:36,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326836522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326836522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682326776517, max timestamp=1682326776524Num uniques 1057; 2023-04-24 08:59:36,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-04-24 08:59:36,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682326776517, max timestamp=1682326776524Num uniques 1057; 2023-04-24 08:59:36,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1682326776517, max timestamp=1682326776524Num uniques 1057; 2023-04-24 08:59:36,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-04-24 08:59:36,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-04-24 08:59:36,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1682326776536, max timestamp=1682326776543Num uniques 1066; 2023-04-24 08:59:36,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-24 08:59:36,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1682326776536, max timestamp=1682326776543Num uniques 1066; 2023-04-24 08:59:36,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-24 08:59:36,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1682326776536, max timestamp=1682326776543Num uniques 1066; 2023-04-24 08:59:36,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-04-24 08:59:36,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326836568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8000, totalHeapSize=18440, min timestamp=1682326776557, max timestamp=1682326776570Num uniques 1075; 2023-04-24 08:59:36,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49220, new segment data size=8000, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-04-24 08:59:36,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8002, totalHeapSize=18440, min timestamp=1682326776557, max timestamp=1682326776570Num uniques 1075; 2023-04-24 08:59:36,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1682326776557, max timestamp=1682326776570Num uniques 1075; 2023-04-24 08:59:36,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49222, new segment data size=8002, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-04-24 08:59:36,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=8010, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-04-24 08:59:36,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1682326776589, max timestamp=1682326776599Num uniques 1034; 2023-04-24 08:59:36,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1682326776589, max timestamp=1682326776599Num uniques 1034; 2023-04-24 08:59:36,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-04-24 08:59:36,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-04-24 08:59:36,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1682326776589, max timestamp=1682326776599Num uniques 1034; 2023-04-24 08:59:36,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-04-24 08:59:36,603 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:36,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:36,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:36,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:36,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:36,606 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:36,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:36,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:36,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,611 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,611 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,611 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:36,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:36,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:36,612 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,612 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:36,612 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:36,612 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:36,612 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:36,619 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,619 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,619 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,619 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:36,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34f6a99d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:36,621 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:36,621 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,627 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:36,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:36,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:36,628 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,628 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:36,636 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,636 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:36,636 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1682326776627, max timestamp=1682326776633Num uniques 1093; 2023-04-24 08:59:36,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1682326776627, max timestamp=1682326776633Num uniques 1093; 2023-04-24 08:59:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1682326776627, max timestamp=1682326776633Num uniques 1093; 2023-04-24 08:59:36,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-24 08:59:36,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-24 08:59:36,636 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:36,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-24 08:59:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:36,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:36,638 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:36,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:36,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:36,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:36,639 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:36,644 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,644 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,644 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,653 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:36,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:36,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,653 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:36,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:36,653 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:36,653 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,654 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,654 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1682326776638, max timestamp=1682326776662Num uniques 1102; 2023-04-24 08:59:36,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1682326776638, max timestamp=1682326776662Num uniques 1102; 2023-04-24 08:59:36,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1682326776638, max timestamp=1682326776662Num uniques 1102; 2023-04-24 08:59:36,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326836660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-24 08:59:36,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-24 08:59:36,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=6960, suffix heap size=142576,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=152 2023-04-24 08:59:36,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1682326776680, max timestamp=1682326776686Num uniques 1011; 2023-04-24 08:59:36,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1682326776680, max timestamp=1682326776686Num uniques 1011; 2023-04-24 08:59:36,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=11946, suffix heap size=133112,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=261 2023-04-24 08:59:36,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1682326776680, max timestamp=1682326776686Num uniques 1011; 2023-04-24 08:59:36,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=11946, suffix heap size=133112,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=261 2023-04-24 08:59:36,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=11946, suffix heap size=133112,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=261 2023-04-24 08:59:36,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/108806ad82f441d7ad46d997589dd5c7 2023-04-24 08:59:36,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/d556a356edd441bf8556da5c7cf6902c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d556a356edd441bf8556da5c7cf6902c 2023-04-24 08:59:36,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d556a356edd441bf8556da5c7cf6902c, entries=200, sequenceid=336, filesize=14.3 K 2023-04-24 08:59:36,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/57c1ceb052a541139f3711756c2e19b8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/57c1ceb052a541139f3711756c2e19b8 2023-04-24 08:59:36,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326836701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1682326776695, max timestamp=1682326776705Num uniques 1120; 2023-04-24 08:59:36,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=12360, suffix heap size=144448,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=270 2023-04-24 08:59:36,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1682326776695, max timestamp=1682326776705Num uniques 1120; 2023-04-24 08:59:36,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1682326776695, max timestamp=1682326776705Num uniques 1120; 2023-04-24 08:59:36,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=12360, suffix heap size=144448,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=270 2023-04-24 08:59:36,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=12360, suffix heap size=144448,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=270 2023-04-24 08:59:36,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/57c1ceb052a541139f3711756c2e19b8, entries=150, sequenceid=336, filesize=11.9 K 2023-04-24 08:59:36,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/108806ad82f441d7ad46d997589dd5c7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/108806ad82f441d7ad46d997589dd5c7 2023-04-24 08:59:36,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/108806ad82f441d7ad46d997589dd5c7, entries=150, sequenceid=336, filesize=11.9 K 2023-04-24 08:59:36,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=60.38 KB/61830 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 844ms, sequenceid=336, compaction requested=true 2023-04-24 08:59:36,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:36,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:36,715 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:36,715 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:36,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:36,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:36,716 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:36,716 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/B is initiating minor compaction (all files) 2023-04-24 08:59:36,716 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/B in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,716 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:36,716 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/0895948cca934beb903fc0deeab89b83, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5415009cd6c64410a317be40e8f803aa, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/57c1ceb052a541139f3711756c2e19b8] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=36.3 K 2023-04-24 08:59:36,717 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/A is initiating minor compaction (all files) 2023-04-24 08:59:36,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:36,717 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=67.09 KB heapSize=175.36 KB 2023-04-24 08:59:36,717 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/A in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,717 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ad1f4c5dfd794139a05795dc942e80f5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/21e4dc3e35de4d1ca6caabb4ef642ee8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d556a356edd441bf8556da5c7cf6902c] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=38.7 K 2023-04-24 08:59:36,717 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 0895948cca934beb903fc0deeab89b83, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682326772820 2023-04-24 08:59:36,717 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting ad1f4c5dfd794139a05795dc942e80f5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682326772820 2023-04-24 08:59:36,717 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5415009cd6c64410a317be40e8f803aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326773132 2023-04-24 08:59:36,717 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 21e4dc3e35de4d1ca6caabb4ef642ee8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326773132 2023-04-24 08:59:36,718 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 57c1ceb052a541139f3711756c2e19b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682326775261 2023-04-24 08:59:36,718 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting d556a356edd441bf8556da5c7cf6902c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682326775261 2023-04-24 08:59:36,718 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:36,718 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:36,718 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:36,718 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:36,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:36,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:36,724 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#A#compaction#235 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:36,724 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#B#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:36,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326836745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326836745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,754 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:36,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:36,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326836750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:36,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:36,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:36,756 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:36,757 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,757 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:36,757 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:36,757 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,757 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,761 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,761 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,761 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,761 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:36,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:36,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:36,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:36,762 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:36,762 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,762 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:36,768 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:36,768 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,768 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:36,768 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:36,768 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:36,768 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,768 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:36,768 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:36,768 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,768 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:36,768 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:36,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326836766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,778 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,778 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,778 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:36,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3594d79f[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:36,779 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:36,779 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,786 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:36,786 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,786 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:36,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:36,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:36,794 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:36,794 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,794 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,794 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:36,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:36,796 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:36,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:36,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:36,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:36,797 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:36,802 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,802 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,803 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,804 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326836807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:36,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,811 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:36,811 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,812 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:36,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326836851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326836851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326836856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,912 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:36,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:36,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:36,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:36,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:36,914 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:36,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:36,915 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:36,915 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,915 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,919 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,919 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,919 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,919 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:36,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:36,920 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:36,921 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,921 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:36,921 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:36,921 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:36,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:36,927 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,927 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,928 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,928 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:36,929 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:36,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c4e1050[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:36,929 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,936 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,936 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:36,936 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:36,936 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:36,936 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,936 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:36,944 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:36,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,944 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,944 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:36,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:36,946 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:36,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:36,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:36,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:36,947 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:36,952 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:36,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,954 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,961 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:36,961 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:36,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:36,961 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:36,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:36,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:36,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:36,961 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,961 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:36,961 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:36,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:36,961 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:36,961 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:36,962 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,962 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:36,962 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:36,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:36,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:36,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326836970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326837010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326837053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326837054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326837058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,062 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:37,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:37,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:37,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:37,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:37,064 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:37,064 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,064 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:37,064 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:37,065 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,065 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,069 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,069 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,069 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,070 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:37,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:37,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:37,070 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:37,071 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,071 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:37,071 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:37,071 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:37,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:37,077 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,078 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,078 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,078 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:37,079 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:37,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6df319e0[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:37,079 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:37,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:37,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:37,086 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:37,094 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:37,094 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:37,094 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,094 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:37,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:37,096 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:37,096 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:37,096 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:37,096 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:37,096 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:37,102 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,102 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:37,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,103 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:37,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,104 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,129 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:37,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:37,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:37,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:37,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:37,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,130 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:37,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,130 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:37,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:37,146 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/03fa1f5080c94fdf81709460b1d7436f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/03fa1f5080c94fdf81709460b1d7436f 2023-04-24 08:59:37,147 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/c83264a650eb45389a2b5660a35dd47e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/c83264a650eb45389a2b5660a35dd47e 2023-04-24 08:59:37,150 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/A of 9ccb4fdcc4ce1f47af3dea271de2f091 into 03fa1f5080c94fdf81709460b1d7436f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:37,150 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/B of 9ccb4fdcc4ce1f47af3dea271de2f091 into c83264a650eb45389a2b5660a35dd47e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:37,150 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:37,150 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:37,150 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/A, priority=13, startTime=1682326776715; duration=0sec 2023-04-24 08:59:37,151 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/B, priority=13, startTime=1682326776715; duration=0sec 2023-04-24 08:59:37,151 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:37,151 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:37,151 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 08:59:37,151 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 08:59:37,151 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/C is initiating minor compaction (all files) 2023-04-24 08:59:37,152 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/C in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,152 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e055d1d33523416fa7d40a1e14c7c802, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/b80a2a365d5440c2a6a89c0b215cd450, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/108806ad82f441d7ad46d997589dd5c7] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=36.3 K 2023-04-24 08:59:37,152 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e055d1d33523416fa7d40a1e14c7c802, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682326772820 2023-04-24 08:59:37,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/fbb79557e76643479228df5c4f9f764f 2023-04-24 08:59:37,153 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting b80a2a365d5440c2a6a89c0b215cd450, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326773132 2023-04-24 08:59:37,153 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 108806ad82f441d7ad46d997589dd5c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682326775261 2023-04-24 08:59:37,174 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#C#compaction#237 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:37,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/83ed9b0adc4e4b97bfae57553e3c658f 2023-04-24 08:59:37,204 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/1d21c6b2777e41b8b1081cc079379bd6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1d21c6b2777e41b8b1081cc079379bd6 2023-04-24 08:59:37,210 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/C of 9ccb4fdcc4ce1f47af3dea271de2f091 into 1d21c6b2777e41b8b1081cc079379bd6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:37,210 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:37,210 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/C, priority=13, startTime=1682326776715; duration=0sec 2023-04-24 08:59:37,210 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:37,230 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:37,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:37,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:37,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:37,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:37,233 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:37,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:37,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:37,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,243 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,243 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:37,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:37,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:37,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:37,245 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,245 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:37,245 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:37,245 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:37,245 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:37,253 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,253 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,253 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,253 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:37,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc8ef82[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:37,254 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:37,254 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,261 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,261 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:37,261 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:37,261 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:37,261 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,261 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:37,269 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,269 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,269 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:37,269 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:37,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:37,271 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:37,271 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:37,271 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:37,271 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:37,271 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:37,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326837273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,277 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,277 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,278 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:37,278 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,279 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:37,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,286 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:37,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:37,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:37,286 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,286 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:37,286 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:37,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326837315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326837356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326837359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326837362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,386 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:37,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:37,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:37,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:37,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:37,392 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:37,392 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,392 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:37,392 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:37,393 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,393 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,403 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:37,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:37,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:37,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:37,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:37,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:37,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:37,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:37,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:37,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:37,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:37,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:37,404 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:37,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:37,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:37,411 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,411 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,411 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,411 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:37,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:37,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f73f5ab[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:37,413 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:37,413 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,419 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,419 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:37,419 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,420 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:37,419 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:37,420 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:37,428 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:37,428 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,428 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,428 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:37,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:37,430 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:37,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:37,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:37,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:37,430 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:37,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:37,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:37,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,437 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,438 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:37,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:37,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:37,444 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:37,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:37,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,445 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,445 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,445 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:37,445 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:37,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:37,545 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:37,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:37,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:37,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:37,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:37,554 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:37,554 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,554 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:37,554 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:37,556 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,556 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,561 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,561 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,561 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,561 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:37,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:37,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:37,562 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:37,562 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,562 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:37,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:37,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:37,563 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,563 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:37,563 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:37,563 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:37,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:37,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:37,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:37,569 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,569 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:37,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4303a685[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:37,571 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:37,571 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,578 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,578 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:37,578 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:37,578 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:37,578 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,578 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:37,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:37,586 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,586 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:37,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:37,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:37,591 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:37,591 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:37,591 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:37,591 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:37,591 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:37,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:37,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:37,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:37,602 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,603 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:37,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:37,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:37,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,611 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:37,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:37,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,611 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:37,612 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:37,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:37,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:37,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/ee6731e3c227460dbd0dc586a8c09982 2023-04-24 08:59:37,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/fbb79557e76643479228df5c4f9f764f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/fbb79557e76643479228df5c4f9f764f 2023-04-24 08:59:37,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/fbb79557e76643479228df5c4f9f764f, entries=250, sequenceid=861, filesize=16.7 K 2023-04-24 08:59:37,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/83ed9b0adc4e4b97bfae57553e3c658f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/83ed9b0adc4e4b97bfae57553e3c658f 2023-04-24 08:59:37,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/83ed9b0adc4e4b97bfae57553e3c658f, entries=150, sequenceid=861, filesize=11.9 K 2023-04-24 08:59:37,633 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/ee6731e3c227460dbd0dc586a8c09982 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/ee6731e3c227460dbd0dc586a8c09982 2023-04-24 08:59:37,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/ee6731e3c227460dbd0dc586a8c09982, entries=150, sequenceid=861, filesize=11.9 K 2023-04-24 08:59:37,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~215.16 KB/220320, currentSize=100.63 KB/103050 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 921ms, sequenceid=861, compaction requested=false 2023-04-24 08:59:37,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:37,712 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:37,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:37,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:37,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:37,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:37,715 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:37,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:37,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:37,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:37,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:37,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,720 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:37,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:37,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:37,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:37,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:37,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:37,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:37,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:37,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:37,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:37,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:37,721 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 08:59:37,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:37,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:37,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:37,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:37,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:37,721 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:37,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:37,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:37,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326837805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326837820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326837862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326837863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326837866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:37,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:37,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326837908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:38,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326838112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,127 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=882 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/7c05d610a6dc4035b8d068dde3b73314 2023-04-24 08:59:38,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:38,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326838415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,536 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=882 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/d8ccc4a054b74da4abbd642e29c49fe4 2023-04-24 08:59:38,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:38,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326838832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:38,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326838865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:38,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326838873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:38,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326838876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:38,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326838922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,944 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=882 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/bf9c4c822b16456d8a1231010ec92184 2023-04-24 08:59:38,948 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/7c05d610a6dc4035b8d068dde3b73314 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/7c05d610a6dc4035b8d068dde3b73314 2023-04-24 08:59:38,957 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/7c05d610a6dc4035b8d068dde3b73314, entries=150, sequenceid=882, filesize=11.9 K 2023-04-24 08:59:38,958 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/d8ccc4a054b74da4abbd642e29c49fe4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d8ccc4a054b74da4abbd642e29c49fe4 2023-04-24 08:59:38,963 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d8ccc4a054b74da4abbd642e29c49fe4, entries=150, sequenceid=882, filesize=11.9 K 2023-04-24 08:59:38,964 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/bf9c4c822b16456d8a1231010ec92184 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bf9c4c822b16456d8a1231010ec92184 2023-04-24 08:59:38,973 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bf9c4c822b16456d8a1231010ec92184, entries=150, sequenceid=882, filesize=11.9 K 2023-04-24 08:59:38,974 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 1253ms, sequenceid=882, compaction requested=true 2023-04-24 08:59:38,974 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:38,974 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:38,975 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:38,975 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:39,012 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:39,012 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:39,012 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:39,012 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:39,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:39,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ca9b1c[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:39,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:39,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:39,036 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:39,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:39,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:39,036 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:39,036 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:39,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:39,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:39,077 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:39,077 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,077 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:39,077 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:39,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:39,079 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:39,079 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:39,079 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:39,079 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:39,079 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:39,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:39,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:39,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:39,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:39,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:39,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:39,087 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:39,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:39,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:39,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:39,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:39,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:39,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:39,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:39,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:39,120 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:39,120 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:39,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:39,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,120 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:39,120 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:39,120 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:39,220 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:39,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:39,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:39,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:39,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:39,223 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:39,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:39,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:39,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:39,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:39,244 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:39,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:39,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:39,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:39,244 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:39,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:39,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:39,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:39,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:39,252 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:39,252 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:39,252 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:39,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:39,264 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:39,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:39,264 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:39,264 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-24 08:59:39,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:39,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:39,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:39,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:39,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:39,265 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:39,691 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/1485afb3a299495c8388de8aaa61d8de 2023-04-24 08:59:39,700 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/753518d8cd1841a5b4e77d29f7cb1c81 2023-04-24 08:59:39,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:39,934 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:39,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:39,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326839983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326840086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,110 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=896 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/7d9e613f7c06404e80068eae99a5b680 2023-04-24 08:59:40,113 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/1485afb3a299495c8388de8aaa61d8de as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1485afb3a299495c8388de8aaa61d8de 2023-04-24 08:59:40,116 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1485afb3a299495c8388de8aaa61d8de, entries=150, sequenceid=896, filesize=11.9 K 2023-04-24 08:59:40,117 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/753518d8cd1841a5b4e77d29f7cb1c81 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/753518d8cd1841a5b4e77d29f7cb1c81 2023-04-24 08:59:40,120 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/753518d8cd1841a5b4e77d29f7cb1c81, entries=150, sequenceid=896, filesize=11.9 K 2023-04-24 08:59:40,121 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/7d9e613f7c06404e80068eae99a5b680 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/7d9e613f7c06404e80068eae99a5b680 2023-04-24 08:59:40,124 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/7d9e613f7c06404e80068eae99a5b680, entries=150, sequenceid=896, filesize=11.9 K 2023-04-24 08:59:40,125 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 861ms, sequenceid=896, compaction requested=true 2023-04-24 08:59:40,125 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:40,125 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:40,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:40,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:40,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:40,125 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:40,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:40,136 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:40,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:40,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:40,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:40,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd5d6b8[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:40,137 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:40,137 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:40,144 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:40,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:40,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:40,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:40,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:40,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:40,144 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,144 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:40,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:40,152 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:40,153 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:40,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:40,156 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:40,156 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:40,156 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:40,156 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:40,156 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:40,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:40,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,161 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:40,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:40,162 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:40,169 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:40,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:40,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:40,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:40,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:40,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:40,170 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:40,170 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:40,270 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:40,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:40,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:40,274 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:40,274 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:40,274 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:40,274 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:40,274 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,274 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:40,277 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:40,277 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:40,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:40,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:40,279 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,279 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:40,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:40,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:40,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:40,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:40,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:40,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:40,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:40,280 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:40,280 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. started... 2023-04-24 08:59:40,281 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-24 08:59:40,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:40,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:40,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:40,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:40,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:40,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:40,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:40,290 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. as already flushing 2023-04-24 08:59:40,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326840321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326840424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,467 DEBUG [Thread-1438] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38204078 to 127.0.0.1:63978 2023-04-24 08:59:40,467 DEBUG [Thread-1430] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07742ca6 to 127.0.0.1:63978 2023-04-24 08:59:40,467 DEBUG [Thread-1430] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:40,467 DEBUG [Thread-1436] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6635f17a to 127.0.0.1:63978 2023-04-24 08:59:40,467 DEBUG [Thread-1436] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:40,467 DEBUG [Thread-1438] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:40,470 DEBUG [Thread-1434] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12f583c6 to 127.0.0.1:63978 2023-04-24 08:59:40,470 DEBUG [Thread-1434] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:40,470 DEBUG [Thread-1432] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x353e11a8 to 127.0.0.1:63978 2023-04-24 08:59:40,470 DEBUG [Thread-1432] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:40,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326840630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,695 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=914 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/17a423cf4dea489f9c70777712fb359b 2023-04-24 08:59:40,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34670 deadline: 1682326840852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,853 DEBUG [Thread-1421] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:59:40,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34652 deadline: 1682326840867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,868 DEBUG [Thread-1425] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4123 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:59:40,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34630 deadline: 1682326840886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34680 deadline: 1682326840886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:40,888 DEBUG [Thread-1427] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:59:40,888 DEBUG [Thread-1419] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 08:59:40,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:40,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326840933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,115 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=914 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/397e6fb2681e4318a8db3c938a9bf614 2023-04-24 08:59:41,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:41,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34616 deadline: 1682326841437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,524 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=914 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/0ae42d51ffbc49249b4274e3e5df988d 2023-04-24 08:59:41,528 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/17a423cf4dea489f9c70777712fb359b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/17a423cf4dea489f9c70777712fb359b 2023-04-24 08:59:41,531 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/17a423cf4dea489f9c70777712fb359b, entries=150, sequenceid=914, filesize=11.9 K 2023-04-24 08:59:41,532 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/397e6fb2681e4318a8db3c938a9bf614 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/397e6fb2681e4318a8db3c938a9bf614 2023-04-24 08:59:41,535 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/397e6fb2681e4318a8db3c938a9bf614, entries=150, sequenceid=914, filesize=11.9 K 2023-04-24 08:59:41,536 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/0ae42d51ffbc49249b4274e3e5df988d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/0ae42d51ffbc49249b4274e3e5df988d 2023-04-24 08:59:41,539 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/0ae42d51ffbc49249b4274e3e5df988d, entries=150, sequenceid=914, filesize=11.9 K 2023-04-24 08:59:41,540 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 1259ms, sequenceid=914, compaction requested=true 2023-04-24 08:59:41,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:41,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:41,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:41,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:41,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:41,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:41,552 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,552 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:41,552 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:41,552 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:41,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:41,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7debb604[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:41,554 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:41,554 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:41,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:41,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:41,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:41,569 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:41,569 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:41,569 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,569 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:41,569 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:41,585 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:41,585 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,585 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:41,585 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:41,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:41,588 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:41,588 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:41,588 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:41,588 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:41,588 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:41,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:41,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:41,602 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:41,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:41,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:41,603 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:41,604 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:41,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:41,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,618 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:41,619 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:41,618 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:41,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:41,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:41,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:41,619 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:41,619 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:41,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:41,619 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:41,619 DEBUG [Thread-1429] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:41,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:41,619 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:41,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:41,619 DEBUG [Thread-1429] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:41,619 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:41,620 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:41,620 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:41,720 DEBUG [Thread-1429] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:41,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:42,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:42,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-24 08:59:42,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:42,444 DEBUG [Thread-1423] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24e1e878 to 127.0.0.1:63978 2023-04-24 08:59:42,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:42,444 DEBUG [Thread-1423] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:42,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:42,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:42,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:42,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:42,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/a42be49ebba845cdb32d3498858d2bfc 2023-04-24 08:59:43,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/1f5d369564ae4b7d9071587531fd96d5 2023-04-24 08:59:43,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/e89d702bbaf0477c97ccd8f5489f5885 2023-04-24 08:59:43,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/a42be49ebba845cdb32d3498858d2bfc as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a42be49ebba845cdb32d3498858d2bfc 2023-04-24 08:59:43,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a42be49ebba845cdb32d3498858d2bfc, entries=150, sequenceid=929, filesize=11.9 K 2023-04-24 08:59:43,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/1f5d369564ae4b7d9071587531fd96d5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1f5d369564ae4b7d9071587531fd96d5 2023-04-24 08:59:43,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1f5d369564ae4b7d9071587531fd96d5, entries=150, sequenceid=929, filesize=11.9 K 2023-04-24 08:59:43,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/e89d702bbaf0477c97ccd8f5489f5885 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e89d702bbaf0477c97ccd8f5489f5885 2023-04-24 08:59:43,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e89d702bbaf0477c97ccd8f5489f5885, entries=150, sequenceid=929, filesize=11.9 K 2023-04-24 08:59:43,707 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 1263ms, sequenceid=929, compaction requested=true 2023-04-24 08:59:43,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:43,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:43,708 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 08:59:43,708 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 08:59:43,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:43,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:43,709 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 78678 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 08:59:43,709 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/A is initiating minor compaction (all files) 2023-04-24 08:59:43,710 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 08:59:43,710 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/A in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:43,710 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/B is initiating minor compaction (all files) 2023-04-24 08:59:43,710 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/03fa1f5080c94fdf81709460b1d7436f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/fbb79557e76643479228df5c4f9f764f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/7c05d610a6dc4035b8d068dde3b73314, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1485afb3a299495c8388de8aaa61d8de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/17a423cf4dea489f9c70777712fb359b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a42be49ebba845cdb32d3498858d2bfc] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=76.8 K 2023-04-24 08:59:43,710 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/B in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:43,710 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/c83264a650eb45389a2b5660a35dd47e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/83ed9b0adc4e4b97bfae57553e3c658f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d8ccc4a054b74da4abbd642e29c49fe4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/753518d8cd1841a5b4e77d29f7cb1c81, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/397e6fb2681e4318a8db3c938a9bf614, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1f5d369564ae4b7d9071587531fd96d5] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=72.1 K 2023-04-24 08:59:43,710 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 03fa1f5080c94fdf81709460b1d7436f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682326775261 2023-04-24 08:59:43,710 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c83264a650eb45389a2b5660a35dd47e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682326775261 2023-04-24 08:59:43,710 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting fbb79557e76643479228df5c4f9f764f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1682326776709 2023-04-24 08:59:43,710 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 83ed9b0adc4e4b97bfae57553e3c658f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1682326776709 2023-04-24 08:59:43,711 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 7c05d610a6dc4035b8d068dde3b73314, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1682326776738 2023-04-24 08:59:43,711 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d8ccc4a054b74da4abbd642e29c49fe4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1682326776738 2023-04-24 08:59:43,711 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1485afb3a299495c8388de8aaa61d8de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=896, earliestPutTs=1682326777795 2023-04-24 08:59:43,711 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 753518d8cd1841a5b4e77d29f7cb1c81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=896, earliestPutTs=1682326777795 2023-04-24 08:59:43,711 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 17a423cf4dea489f9c70777712fb359b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682326779967 2023-04-24 08:59:43,711 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 397e6fb2681e4318a8db3c938a9bf614, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682326779967 2023-04-24 08:59:43,712 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting a42be49ebba845cdb32d3498858d2bfc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1682326780302 2023-04-24 08:59:43,712 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 1f5d369564ae4b7d9071587531fd96d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1682326780302 2023-04-24 08:59:43,724 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#B#compaction#252 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:43,724 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#A#compaction#253 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:44,136 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/6ebc121c32624956ab7e267ae227b988 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/6ebc121c32624956ab7e267ae227b988 2023-04-24 08:59:44,136 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/e54289221334489ba9f75457c2340db9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e54289221334489ba9f75457c2340db9 2023-04-24 08:59:44,144 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/A of 9ccb4fdcc4ce1f47af3dea271de2f091 into e54289221334489ba9f75457c2340db9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:44,144 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/B of 9ccb4fdcc4ce1f47af3dea271de2f091 into 6ebc121c32624956ab7e267ae227b988(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:44,144 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:44,144 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:44,144 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/A, priority=10, startTime=1682326783708; duration=0sec 2023-04-24 08:59:44,144 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/B, priority=10, startTime=1682326783708; duration=0sec 2023-04-24 08:59:44,144 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:44,144 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:44,144 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 08:59:44,146 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 08:59:44,146 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 9ccb4fdcc4ce1f47af3dea271de2f091/C is initiating minor compaction (all files) 2023-04-24 08:59:44,146 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ccb4fdcc4ce1f47af3dea271de2f091/C in TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:44,146 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1d21c6b2777e41b8b1081cc079379bd6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/ee6731e3c227460dbd0dc586a8c09982, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bf9c4c822b16456d8a1231010ec92184, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/7d9e613f7c06404e80068eae99a5b680, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/0ae42d51ffbc49249b4274e3e5df988d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e89d702bbaf0477c97ccd8f5489f5885] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp, totalSize=72.1 K 2023-04-24 08:59:44,147 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1d21c6b2777e41b8b1081cc079379bd6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1682326775261 2023-04-24 08:59:44,147 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting ee6731e3c227460dbd0dc586a8c09982, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1682326776709 2023-04-24 08:59:44,147 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting bf9c4c822b16456d8a1231010ec92184, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=882, earliestPutTs=1682326776738 2023-04-24 08:59:44,148 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 7d9e613f7c06404e80068eae99a5b680, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=896, earliestPutTs=1682326777795 2023-04-24 08:59:44,148 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 0ae42d51ffbc49249b4274e3e5df988d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=914, earliestPutTs=1682326779967 2023-04-24 08:59:44,148 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e89d702bbaf0477c97ccd8f5489f5885, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1682326780302 2023-04-24 08:59:44,158 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ccb4fdcc4ce1f47af3dea271de2f091#C#compaction#254 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:44,567 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/693e6fed9ad0434f99b15138b967b8a9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/693e6fed9ad0434f99b15138b967b8a9 2023-04-24 08:59:44,573 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9ccb4fdcc4ce1f47af3dea271de2f091/C of 9ccb4fdcc4ce1f47af3dea271de2f091 into 693e6fed9ad0434f99b15138b967b8a9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:44,573 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:44,573 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091., storeName=9ccb4fdcc4ce1f47af3dea271de2f091/C, priority=10, startTime=1682326783708; duration=0sec 2023-04-24 08:59:44,573 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:44,866 DEBUG [Thread-1421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f2b45f8 to 127.0.0.1:63978 2023-04-24 08:59:44,866 DEBUG [Thread-1421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:44,899 DEBUG [Thread-1425] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e9921a7 to 127.0.0.1:63978 2023-04-24 08:59:44,899 DEBUG [Thread-1425] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:44,907 DEBUG [Thread-1427] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ffcf146 to 127.0.0.1:63978 2023-04-24 08:59:44,907 DEBUG [Thread-1427] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:44,930 DEBUG [Thread-1419] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2135e03b to 127.0.0.1:63978 2023-04-24 08:59:44,930 DEBUG [Thread-1419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:44,930 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 08:59:44,930 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 236 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 138 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 127 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5254 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5243 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5386 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5229 2023-04-24 08:59:44,931 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5267 2023-04-24 08:59:44,932 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 08:59:44,932 INFO [Listener at localhost.localdomain/44291] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 08:59:44,932 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73ef029c to 127.0.0.1:63978 2023-04-24 08:59:44,932 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 08:59:44,933 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 08:59:44,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-24 08:59:44,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:44,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 08:59:44,939 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326784939"}]},"ts":"1682326784939"} 2023-04-24 08:59:44,940 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 08:59:44,978 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 08:59:44,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ccb4fdcc4ce1f47af3dea271de2f091, UNASSIGN}] 2023-04-24 08:59:44,982 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ccb4fdcc4ce1f47af3dea271de2f091, UNASSIGN 2023-04-24 08:59:44,984 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=9ccb4fdcc4ce1f47af3dea271de2f091, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:44,985 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326784984"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326784984"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326784984"}]},"ts":"1682326784984"} 2023-04-24 08:59:44,987 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:59:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 08:59:45,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:45,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 9ccb4fdcc4ce1f47af3dea271de2f091, disabling compactions & flushes 2023-04-24 08:59:45,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:45,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:45,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. after waiting 0 ms 2023-04-24 08:59:45,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:45,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 9ccb4fdcc4ce1f47af3dea271de2f091 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 08:59:45,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=A 2023-04-24 08:59:45,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:45,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=B 2023-04-24 08:59:45,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:45,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ccb4fdcc4ce1f47af3dea271de2f091, store=C 2023-04-24 08:59:45,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:45,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 08:59:45,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 08:59:45,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=939 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/e1000f503a9745b5b31b2fe9bd627f69 2023-04-24 08:59:45,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=939 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/fb2f7b4694ea44a5bec1e44263409244 2023-04-24 08:59:46,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 08:59:46,389 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=939 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/a20245f5392e4bc6aeb9e528f8f1f0b0 2023-04-24 08:59:46,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/A/e1000f503a9745b5b31b2fe9bd627f69 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e1000f503a9745b5b31b2fe9bd627f69 2023-04-24 08:59:46,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e1000f503a9745b5b31b2fe9bd627f69, entries=150, sequenceid=939, filesize=11.9 K 2023-04-24 08:59:46,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/B/fb2f7b4694ea44a5bec1e44263409244 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/fb2f7b4694ea44a5bec1e44263409244 2023-04-24 08:59:46,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/fb2f7b4694ea44a5bec1e44263409244, entries=150, sequenceid=939, filesize=11.9 K 2023-04-24 08:59:46,413 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/.tmp/C/a20245f5392e4bc6aeb9e528f8f1f0b0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/a20245f5392e4bc6aeb9e528f8f1f0b0 2023-04-24 08:59:46,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/a20245f5392e4bc6aeb9e528f8f1f0b0, entries=150, sequenceid=939, filesize=11.9 K 2023-04-24 08:59:46,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 9ccb4fdcc4ce1f47af3dea271de2f091 in 1278ms, sequenceid=939, compaction requested=false 2023-04-24 08:59:46,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/4e07db666a3a466892c6699f9a2f11f0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0c08942549414627aa9b5b2d018933e8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ce958721b479466c92bacb232d899735, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/3751e998818e4ddeb364a0814bb87ac1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e00be84ca8414ca0b1720f92efb12649, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/407a71be480240488272e9982a51713a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/b6918d79eea447da9f0d57310a1b48f6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e364409c6fc24917b7eb844aaa56c15b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a6d44802fdc34620b398f02f6879e555, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1205da37395c459e81e86161597fb2e9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/de5d66f6435f4f2f915923027f94a94d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/339c45f18d4d473bad1e32684707acd1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/75638e2eac734071badf5bd7a51a0e23, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/dc8ec1ed6c56478d83e6c10a4180c940, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0204b54886364186b4fef08e20e8bf56, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d69be73e0bee4054ab1601c521983bbd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/f30c68a4e8aa40f7b1d2f1076882aa01, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ad1f4c5dfd794139a05795dc942e80f5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/21e4dc3e35de4d1ca6caabb4ef642ee8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d556a356edd441bf8556da5c7cf6902c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/03fa1f5080c94fdf81709460b1d7436f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/fbb79557e76643479228df5c4f9f764f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/7c05d610a6dc4035b8d068dde3b73314, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1485afb3a299495c8388de8aaa61d8de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/17a423cf4dea489f9c70777712fb359b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a42be49ebba845cdb32d3498858d2bfc] to archive 2023-04-24 08:59:46,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:59:46,444 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/4e07db666a3a466892c6699f9a2f11f0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/4e07db666a3a466892c6699f9a2f11f0 2023-04-24 08:59:46,445 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0c08942549414627aa9b5b2d018933e8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0c08942549414627aa9b5b2d018933e8 2023-04-24 08:59:46,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ce958721b479466c92bacb232d899735 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ce958721b479466c92bacb232d899735 2023-04-24 08:59:46,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/3751e998818e4ddeb364a0814bb87ac1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/3751e998818e4ddeb364a0814bb87ac1 2023-04-24 08:59:46,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e00be84ca8414ca0b1720f92efb12649 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e00be84ca8414ca0b1720f92efb12649 2023-04-24 08:59:46,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/407a71be480240488272e9982a51713a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/407a71be480240488272e9982a51713a 2023-04-24 08:59:46,451 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/b6918d79eea447da9f0d57310a1b48f6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/b6918d79eea447da9f0d57310a1b48f6 2023-04-24 08:59:46,452 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e364409c6fc24917b7eb844aaa56c15b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e364409c6fc24917b7eb844aaa56c15b 2023-04-24 08:59:46,453 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a6d44802fdc34620b398f02f6879e555 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a6d44802fdc34620b398f02f6879e555 2023-04-24 08:59:46,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1205da37395c459e81e86161597fb2e9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1205da37395c459e81e86161597fb2e9 2023-04-24 08:59:46,455 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/de5d66f6435f4f2f915923027f94a94d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/de5d66f6435f4f2f915923027f94a94d 2023-04-24 08:59:46,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/339c45f18d4d473bad1e32684707acd1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/339c45f18d4d473bad1e32684707acd1 2023-04-24 08:59:46,457 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/75638e2eac734071badf5bd7a51a0e23 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/75638e2eac734071badf5bd7a51a0e23 2023-04-24 08:59:46,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/dc8ec1ed6c56478d83e6c10a4180c940 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/dc8ec1ed6c56478d83e6c10a4180c940 2023-04-24 08:59:46,459 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0204b54886364186b4fef08e20e8bf56 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/0204b54886364186b4fef08e20e8bf56 2023-04-24 08:59:46,459 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d69be73e0bee4054ab1601c521983bbd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d69be73e0bee4054ab1601c521983bbd 2023-04-24 08:59:46,461 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/f30c68a4e8aa40f7b1d2f1076882aa01 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/f30c68a4e8aa40f7b1d2f1076882aa01 2023-04-24 08:59:46,462 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ad1f4c5dfd794139a05795dc942e80f5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/ad1f4c5dfd794139a05795dc942e80f5 2023-04-24 08:59:46,463 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/21e4dc3e35de4d1ca6caabb4ef642ee8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/21e4dc3e35de4d1ca6caabb4ef642ee8 2023-04-24 08:59:46,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d556a356edd441bf8556da5c7cf6902c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/d556a356edd441bf8556da5c7cf6902c 2023-04-24 08:59:46,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/03fa1f5080c94fdf81709460b1d7436f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/03fa1f5080c94fdf81709460b1d7436f 2023-04-24 08:59:46,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/fbb79557e76643479228df5c4f9f764f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/fbb79557e76643479228df5c4f9f764f 2023-04-24 08:59:46,467 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/7c05d610a6dc4035b8d068dde3b73314 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/7c05d610a6dc4035b8d068dde3b73314 2023-04-24 08:59:46,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1485afb3a299495c8388de8aaa61d8de to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/1485afb3a299495c8388de8aaa61d8de 2023-04-24 08:59:46,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/17a423cf4dea489f9c70777712fb359b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/17a423cf4dea489f9c70777712fb359b 2023-04-24 08:59:46,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a42be49ebba845cdb32d3498858d2bfc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/a42be49ebba845cdb32d3498858d2bfc 2023-04-24 08:59:46,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/a6947aabedca41b89c50c3de5c95a3ad, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d97b333667304f3fab8037b1a0db4ded, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/e8ffedaca6bf4defaad373cdeeeac4ab, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d0c8c7dbbe544043a40e8f32d7da2a70, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/25bf4d3343054b1d8644dd0d63e9fa28, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/869462c969e74c3e9d8c195ae60466ac, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/28ac2f9419744260aa85e9dc8b6a3a60, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/701149c336104e1595fec32e32e98e4d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/bc7a6bf13b9347e893374b444f0b2afe, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d445a287291c42149ffba3125b01515f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/f4f53b136ea4481993ea8a8d82b625e9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5a3e7fad4f3d46bca80bc971b3591e16, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5d985f97f44340c3a570a8684db8cd28, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3f786becb11a4db7b41d22a425701c54, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1d5d3386dc5649f9b1f101c714654a47, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3cd274790a954dafb797ed8ce07330e7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/0895948cca934beb903fc0deeab89b83, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/43faf627ac1d4aecaf32370295ebe53c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5415009cd6c64410a317be40e8f803aa, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/c83264a650eb45389a2b5660a35dd47e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/57c1ceb052a541139f3711756c2e19b8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/83ed9b0adc4e4b97bfae57553e3c658f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d8ccc4a054b74da4abbd642e29c49fe4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/753518d8cd1841a5b4e77d29f7cb1c81, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/397e6fb2681e4318a8db3c938a9bf614, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1f5d369564ae4b7d9071587531fd96d5] to archive 2023-04-24 08:59:46,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:59:46,475 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/a6947aabedca41b89c50c3de5c95a3ad to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/a6947aabedca41b89c50c3de5c95a3ad 2023-04-24 08:59:46,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d97b333667304f3fab8037b1a0db4ded to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d97b333667304f3fab8037b1a0db4ded 2023-04-24 08:59:46,477 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/e8ffedaca6bf4defaad373cdeeeac4ab to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/e8ffedaca6bf4defaad373cdeeeac4ab 2023-04-24 08:59:46,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d0c8c7dbbe544043a40e8f32d7da2a70 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d0c8c7dbbe544043a40e8f32d7da2a70 2023-04-24 08:59:46,479 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/25bf4d3343054b1d8644dd0d63e9fa28 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/25bf4d3343054b1d8644dd0d63e9fa28 2023-04-24 08:59:46,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/869462c969e74c3e9d8c195ae60466ac to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/869462c969e74c3e9d8c195ae60466ac 2023-04-24 08:59:46,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/28ac2f9419744260aa85e9dc8b6a3a60 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/28ac2f9419744260aa85e9dc8b6a3a60 2023-04-24 08:59:46,482 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/701149c336104e1595fec32e32e98e4d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/701149c336104e1595fec32e32e98e4d 2023-04-24 08:59:46,483 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/bc7a6bf13b9347e893374b444f0b2afe to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/bc7a6bf13b9347e893374b444f0b2afe 2023-04-24 08:59:46,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d445a287291c42149ffba3125b01515f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d445a287291c42149ffba3125b01515f 2023-04-24 08:59:46,485 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/f4f53b136ea4481993ea8a8d82b625e9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/f4f53b136ea4481993ea8a8d82b625e9 2023-04-24 08:59:46,486 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5a3e7fad4f3d46bca80bc971b3591e16 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5a3e7fad4f3d46bca80bc971b3591e16 2023-04-24 08:59:46,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5d985f97f44340c3a570a8684db8cd28 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5d985f97f44340c3a570a8684db8cd28 2023-04-24 08:59:46,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3f786becb11a4db7b41d22a425701c54 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3f786becb11a4db7b41d22a425701c54 2023-04-24 08:59:46,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1d5d3386dc5649f9b1f101c714654a47 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1d5d3386dc5649f9b1f101c714654a47 2023-04-24 08:59:46,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3cd274790a954dafb797ed8ce07330e7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/3cd274790a954dafb797ed8ce07330e7 2023-04-24 08:59:46,490 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/0895948cca934beb903fc0deeab89b83 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/0895948cca934beb903fc0deeab89b83 2023-04-24 08:59:46,491 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/43faf627ac1d4aecaf32370295ebe53c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/43faf627ac1d4aecaf32370295ebe53c 2023-04-24 08:59:46,492 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5415009cd6c64410a317be40e8f803aa to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/5415009cd6c64410a317be40e8f803aa 2023-04-24 08:59:46,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/c83264a650eb45389a2b5660a35dd47e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/c83264a650eb45389a2b5660a35dd47e 2023-04-24 08:59:46,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/57c1ceb052a541139f3711756c2e19b8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/57c1ceb052a541139f3711756c2e19b8 2023-04-24 08:59:46,494 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/83ed9b0adc4e4b97bfae57553e3c658f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/83ed9b0adc4e4b97bfae57553e3c658f 2023-04-24 08:59:46,495 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d8ccc4a054b74da4abbd642e29c49fe4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/d8ccc4a054b74da4abbd642e29c49fe4 2023-04-24 08:59:46,495 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/753518d8cd1841a5b4e77d29f7cb1c81 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/753518d8cd1841a5b4e77d29f7cb1c81 2023-04-24 08:59:46,496 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/397e6fb2681e4318a8db3c938a9bf614 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/397e6fb2681e4318a8db3c938a9bf614 2023-04-24 08:59:46,497 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1f5d369564ae4b7d9071587531fd96d5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/1f5d369564ae4b7d9071587531fd96d5 2023-04-24 08:59:46,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e00b3afcb9384ee09c83187a15b55598, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/d7430ed1ebfd439e8148e74be6841d27, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1a31a8dfaee443ebb07c1fdf53efeafd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c4d745402d35466ea2552203fc4e2cc8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/f3649f88bf674d9eb2722cd4ebc711f5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/020a08d7562747f2ad26e78f2ef36875, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/33eb6a2bd8254064a782ee915065c979, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/cba0822c07574d7294aaac685f676a16, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e5593abaac0242aca2c2afeb2718652c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/258a58903cdf4fd9b50f9b7237de6a33, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1cef4573275b49578a9222bda8a46e28, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bcad2a5cb5114187a164c0a73c646143, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/73f6ad27e0304f12a1f496cc9bc91f61, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/61cde1518c9f49dd84ec5dd504d24448, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/3d6011172cc34f7ba3bdf5a466642fd4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c5d2f6b7ce3e4af8892bbdfbc5c89764, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e055d1d33523416fa7d40a1e14c7c802, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/4eeb0a0845894c6aa6be1f77ec7ceba8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/b80a2a365d5440c2a6a89c0b215cd450, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1d21c6b2777e41b8b1081cc079379bd6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/108806ad82f441d7ad46d997589dd5c7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/ee6731e3c227460dbd0dc586a8c09982, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bf9c4c822b16456d8a1231010ec92184, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/7d9e613f7c06404e80068eae99a5b680, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/0ae42d51ffbc49249b4274e3e5df988d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e89d702bbaf0477c97ccd8f5489f5885] to archive 2023-04-24 08:59:46,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 08:59:46,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e00b3afcb9384ee09c83187a15b55598 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e00b3afcb9384ee09c83187a15b55598 2023-04-24 08:59:46,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/d7430ed1ebfd439e8148e74be6841d27 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/d7430ed1ebfd439e8148e74be6841d27 2023-04-24 08:59:46,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1a31a8dfaee443ebb07c1fdf53efeafd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1a31a8dfaee443ebb07c1fdf53efeafd 2023-04-24 08:59:46,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c4d745402d35466ea2552203fc4e2cc8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c4d745402d35466ea2552203fc4e2cc8 2023-04-24 08:59:46,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/f3649f88bf674d9eb2722cd4ebc711f5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/f3649f88bf674d9eb2722cd4ebc711f5 2023-04-24 08:59:46,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/020a08d7562747f2ad26e78f2ef36875 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/020a08d7562747f2ad26e78f2ef36875 2023-04-24 08:59:46,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/33eb6a2bd8254064a782ee915065c979 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/33eb6a2bd8254064a782ee915065c979 2023-04-24 08:59:46,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/cba0822c07574d7294aaac685f676a16 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/cba0822c07574d7294aaac685f676a16 2023-04-24 08:59:46,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e5593abaac0242aca2c2afeb2718652c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e5593abaac0242aca2c2afeb2718652c 2023-04-24 08:59:46,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/258a58903cdf4fd9b50f9b7237de6a33 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/258a58903cdf4fd9b50f9b7237de6a33 2023-04-24 08:59:46,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1cef4573275b49578a9222bda8a46e28 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1cef4573275b49578a9222bda8a46e28 2023-04-24 08:59:46,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bcad2a5cb5114187a164c0a73c646143 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bcad2a5cb5114187a164c0a73c646143 2023-04-24 08:59:46,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/73f6ad27e0304f12a1f496cc9bc91f61 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/73f6ad27e0304f12a1f496cc9bc91f61 2023-04-24 08:59:46,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/61cde1518c9f49dd84ec5dd504d24448 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/61cde1518c9f49dd84ec5dd504d24448 2023-04-24 08:59:46,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/3d6011172cc34f7ba3bdf5a466642fd4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/3d6011172cc34f7ba3bdf5a466642fd4 2023-04-24 08:59:46,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c5d2f6b7ce3e4af8892bbdfbc5c89764 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/c5d2f6b7ce3e4af8892bbdfbc5c89764 2023-04-24 08:59:46,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e055d1d33523416fa7d40a1e14c7c802 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e055d1d33523416fa7d40a1e14c7c802 2023-04-24 08:59:46,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/4eeb0a0845894c6aa6be1f77ec7ceba8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/4eeb0a0845894c6aa6be1f77ec7ceba8 2023-04-24 08:59:46,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/b80a2a365d5440c2a6a89c0b215cd450 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/b80a2a365d5440c2a6a89c0b215cd450 2023-04-24 08:59:46,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1d21c6b2777e41b8b1081cc079379bd6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/1d21c6b2777e41b8b1081cc079379bd6 2023-04-24 08:59:46,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/108806ad82f441d7ad46d997589dd5c7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/108806ad82f441d7ad46d997589dd5c7 2023-04-24 08:59:46,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/ee6731e3c227460dbd0dc586a8c09982 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/ee6731e3c227460dbd0dc586a8c09982 2023-04-24 08:59:46,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bf9c4c822b16456d8a1231010ec92184 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/bf9c4c822b16456d8a1231010ec92184 2023-04-24 08:59:46,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/7d9e613f7c06404e80068eae99a5b680 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/7d9e613f7c06404e80068eae99a5b680 2023-04-24 08:59:46,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/0ae42d51ffbc49249b4274e3e5df988d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/0ae42d51ffbc49249b4274e3e5df988d 2023-04-24 08:59:46,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e89d702bbaf0477c97ccd8f5489f5885 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/e89d702bbaf0477c97ccd8f5489f5885 2023-04-24 08:59:46,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/recovered.edits/942.seqid, newMaxSeqId=942, maxSeqId=1 2023-04-24 08:59:46,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091. 2023-04-24 08:59:46,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 9ccb4fdcc4ce1f47af3dea271de2f091: 2023-04-24 08:59:46,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:46,534 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=9ccb4fdcc4ce1f47af3dea271de2f091, regionState=CLOSED 2023-04-24 08:59:46,535 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326786534"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326786534"}]},"ts":"1682326786534"} 2023-04-24 08:59:46,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-24 08:59:46,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 9ccb4fdcc4ce1f47af3dea271de2f091, server=jenkins-hbase16.apache.org,42837,1682326693550 in 1.5480 sec 2023-04-24 08:59:46,538 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-24 08:59:46,538 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ccb4fdcc4ce1f47af3dea271de2f091, UNASSIGN in 1.5580 sec 2023-04-24 08:59:46,538 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326786538"}]},"ts":"1682326786538"} 2023-04-24 08:59:46,539 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 08:59:46,569 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 08:59:46,571 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6370 sec 2023-04-24 08:59:47,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-24 08:59:47,047 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-24 08:59:47,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-24 08:59:47,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:47,054 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:47,055 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:47,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-24 08:59:47,060 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:47,062 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/recovered.edits] 2023-04-24 08:59:47,065 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e1000f503a9745b5b31b2fe9bd627f69 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e1000f503a9745b5b31b2fe9bd627f69 2023-04-24 08:59:47,067 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e54289221334489ba9f75457c2340db9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/A/e54289221334489ba9f75457c2340db9 2023-04-24 08:59:47,069 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/6ebc121c32624956ab7e267ae227b988 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/6ebc121c32624956ab7e267ae227b988 2023-04-24 08:59:47,070 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/fb2f7b4694ea44a5bec1e44263409244 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/B/fb2f7b4694ea44a5bec1e44263409244 2023-04-24 08:59:47,073 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/693e6fed9ad0434f99b15138b967b8a9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/693e6fed9ad0434f99b15138b967b8a9 2023-04-24 08:59:47,074 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/a20245f5392e4bc6aeb9e528f8f1f0b0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/C/a20245f5392e4bc6aeb9e528f8f1f0b0 2023-04-24 08:59:47,077 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/recovered.edits/942.seqid to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091/recovered.edits/942.seqid 2023-04-24 08:59:47,078 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/9ccb4fdcc4ce1f47af3dea271de2f091 2023-04-24 08:59:47,078 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 08:59:47,080 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:47,081 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 08:59:47,082 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 08:59:47,083 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:47,083 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 08:59:47,083 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682326787083"}]},"ts":"9223372036854775807"} 2023-04-24 08:59:47,084 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 08:59:47,084 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9ccb4fdcc4ce1f47af3dea271de2f091, NAME => 'TestAcidGuarantees,,1682326758242.9ccb4fdcc4ce1f47af3dea271de2f091.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 08:59:47,084 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 08:59:47,085 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682326787085"}]},"ts":"9223372036854775807"} 2023-04-24 08:59:47,087 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 08:59:47,120 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:47,121 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 71 msec 2023-04-24 08:59:47,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-24 08:59:47,157 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-24 08:59:47,169 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 224) - Thread LEAK? -, OpenFileDescriptor=531 (was 538), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=328 (was 352), ProcessCount=168 (was 168), AvailableMemoryMB=8328 (was 8453) 2023-04-24 08:59:47,181 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=328, ProcessCount=168, AvailableMemoryMB=8328 2023-04-24 08:59:47,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 08:59:47,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 08:59:47,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:47,186 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 08:59:47,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-24 08:59:47,187 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 08:59:47,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 08:59:47,188 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:47,188 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 empty. 2023-04-24 08:59:47,189 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:47,189 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 08:59:47,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 08:59:47,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 08:59:47,601 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 08:59:47,605 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d7e0c211118009ab258bf68c7108cae8, NAME => 'TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp 2023-04-24 08:59:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 08:59:48,014 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:59:48,014 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d7e0c211118009ab258bf68c7108cae8, disabling compactions & flushes 2023-04-24 08:59:48,014 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:48,015 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:48,015 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. after waiting 0 ms 2023-04-24 08:59:48,015 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:48,015 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:48,015 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:48,019 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 08:59:48,021 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326788021"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326788021"}]},"ts":"1682326788021"} 2023-04-24 08:59:48,022 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 08:59:48,023 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 08:59:48,023 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326788023"}]},"ts":"1682326788023"} 2023-04-24 08:59:48,024 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 08:59:48,036 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, ASSIGN}] 2023-04-24 08:59:48,037 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, ASSIGN 2023-04-24 08:59:48,037 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=false 2023-04-24 08:59:48,189 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=d7e0c211118009ab258bf68c7108cae8, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:48,189 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326788189"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326788189"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326788189"}]},"ts":"1682326788189"} 2023-04-24 08:59:48,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:59:48,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 08:59:48,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:48,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d7e0c211118009ab258bf68c7108cae8, NAME => 'TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:59:48,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:59:48,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,356 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,359 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A 2023-04-24 08:59:48,359 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A 2023-04-24 08:59:48,359 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:48,360 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7e0c211118009ab258bf68c7108cae8 columnFamilyName A 2023-04-24 08:59:48,361 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(310): Store=d7e0c211118009ab258bf68c7108cae8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:48,361 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,365 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B 2023-04-24 08:59:48,365 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B 2023-04-24 08:59:48,365 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:48,366 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7e0c211118009ab258bf68c7108cae8 columnFamilyName B 2023-04-24 08:59:48,366 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(310): Store=d7e0c211118009ab258bf68c7108cae8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:48,367 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,368 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C 2023-04-24 08:59:48,368 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C 2023-04-24 08:59:48,369 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:48,369 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7e0c211118009ab258bf68c7108cae8 columnFamilyName C 2023-04-24 08:59:48,369 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(310): Store=d7e0c211118009ab258bf68c7108cae8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:48,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:48,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 08:59:48,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:48,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 08:59:48,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened d7e0c211118009ab258bf68c7108cae8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73426270, jitterRate=0.09413668513298035}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 08:59:48,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:48,380 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., pid=37, masterSystemTime=1682326788345 2023-04-24 08:59:48,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:48,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:48,381 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=d7e0c211118009ab258bf68c7108cae8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:48,381 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326788381"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326788381"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326788381"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326788381"}]},"ts":"1682326788381"} 2023-04-24 08:59:48,384 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-24 08:59:48,384 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 in 190 msec 2023-04-24 08:59:48,385 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-24 08:59:48,385 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, ASSIGN in 348 msec 2023-04-24 08:59:48,385 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 08:59:48,385 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326788385"}]},"ts":"1682326788385"} 2023-04-24 08:59:48,386 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 08:59:48,428 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 08:59:48,429 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2440 sec 2023-04-24 08:59:49,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-24 08:59:49,298 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-24 08:59:49,299 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x727ea8af to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:49,338 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f29a773, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:49,341 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:49,346 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:38156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:49,347 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 08:59:49,349 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:46514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 08:59:49,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 08:59:49,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$12(2454): Client=jenkins//188.40.62.62 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 08:59:49,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-24 08:59:49,773 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 08:59:49,776 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 08:59:49,776 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 08:59:49,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-24 08:59:49,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, REOPEN/MOVE}] 2023-04-24 08:59:49,784 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, REOPEN/MOVE 2023-04-24 08:59:49,784 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=d7e0c211118009ab258bf68c7108cae8, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:49,784 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326789784"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326789784"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326789784"}]},"ts":"1682326789784"} 2023-04-24 08:59:49,785 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:59:49,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:49,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing d7e0c211118009ab258bf68c7108cae8, disabling compactions & flushes 2023-04-24 08:59:49,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:49,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:49,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. after waiting 0 ms 2023-04-24 08:59:49,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:49,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-24 08:59:49,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:49,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:49,995 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(3509): Not adding moved region record: d7e0c211118009ab258bf68c7108cae8 to self. 2023-04-24 08:59:49,997 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:49,998 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=d7e0c211118009ab258bf68c7108cae8, regionState=CLOSED 2023-04-24 08:59:49,998 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326789998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326789998"}]},"ts":"1682326789998"} 2023-04-24 08:59:50,000 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-24 08:59:50,000 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 in 214 msec 2023-04-24 08:59:50,001 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=true 2023-04-24 08:59:50,152 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=d7e0c211118009ab258bf68c7108cae8, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,152 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326790151"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326790151"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326790151"}]},"ts":"1682326790151"} 2023-04-24 08:59:50,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 08:59:50,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d7e0c211118009ab258bf68c7108cae8, NAME => 'TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.', STARTKEY => '', ENDKEY => ''} 2023-04-24 08:59:50,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 08:59:50,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,313 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,314 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A 2023-04-24 08:59:50,314 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A 2023-04-24 08:59:50,314 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:50,315 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7e0c211118009ab258bf68c7108cae8 columnFamilyName A 2023-04-24 08:59:50,316 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(310): Store=d7e0c211118009ab258bf68c7108cae8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:50,317 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,318 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B 2023-04-24 08:59:50,318 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B 2023-04-24 08:59:50,318 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:50,318 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7e0c211118009ab258bf68c7108cae8 columnFamilyName B 2023-04-24 08:59:50,319 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(310): Store=d7e0c211118009ab258bf68c7108cae8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:50,319 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,320 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C 2023-04-24 08:59:50,320 DEBUG [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C 2023-04-24 08:59:50,320 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 08:59:50,320 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d7e0c211118009ab258bf68c7108cae8 columnFamilyName C 2023-04-24 08:59:50,321 INFO [StoreOpener-d7e0c211118009ab258bf68c7108cae8-1] regionserver.HStore(310): Store=d7e0c211118009ab258bf68c7108cae8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 08:59:50,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 08:59:50,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened d7e0c211118009ab258bf68c7108cae8; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72111961, jitterRate=0.07455195486545563}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 08:59:50,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:50,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., pid=42, masterSystemTime=1682326790308 2023-04-24 08:59:50,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,331 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=d7e0c211118009ab258bf68c7108cae8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,331 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326790331"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326790331"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326790331"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326790331"}]},"ts":"1682326790331"} 2023-04-24 08:59:50,333 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-24 08:59:50,334 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 in 177 msec 2023-04-24 08:59:50,335 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-24 08:59:50,335 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, REOPEN/MOVE in 551 msec 2023-04-24 08:59:50,336 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-24 08:59:50,337 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 554 msec 2023-04-24 08:59:50,338 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 984 msec 2023-04-24 08:59:50,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-24 08:59:50,340 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x4f54fc1a to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,353 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48681e8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,353 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x5c70a497 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,361 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@342c272f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,362 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x179d9850 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,370 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e90d228, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,370 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x61483370 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,378 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aec6811, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,378 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x4de67ff9 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,386 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d951686, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,386 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x17385c96 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,394 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@525c59d1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,395 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x52829d32 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,403 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78ba718, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,403 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x516a95ee to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,411 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1747f9fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,412 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x3558849b to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,419 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3098b23b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,420 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x24d104d6 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 08:59:50,428 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55dc208b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 08:59:50,432 DEBUG [hconnection-0x497e50f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,438 DEBUG [hconnection-0x10081a8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:50,439 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,442 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52632, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,442 DEBUG [hconnection-0x2394f1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,442 DEBUG [hconnection-0x781f7e97-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,450 DEBUG [hconnection-0x5180df21-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,450 DEBUG [hconnection-0x2c85dcf2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,450 DEBUG [hconnection-0x6fc811bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,448 DEBUG [hconnection-0x511d0782-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,447 DEBUG [hconnection-0x34a3cb4b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,447 DEBUG [hconnection-0x465d7ec1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 08:59:50,451 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:50,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:50,451 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:50,451 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:50,451 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:50,451 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:50,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,453 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,453 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,454 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,455 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,456 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,455 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,456 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,456 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:52694, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 08:59:50,461 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:50,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:50,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:50,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:50,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:50,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:50,468 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:50,468 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:50,468 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:50,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:50,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:50,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:50,474 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 08:59:50,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:50,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:50,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,474 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:50,474 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:50,474 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 08:59:50,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:50,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 08:59:50,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:50,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 08:59:50,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:50,476 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:50,476 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:50,485 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,485 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:50,487 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:50,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@230200b2[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:50,487 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326850487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326850490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326850490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:50,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:50,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:50,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326850491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,495 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,495 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:50,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326850491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:50,503 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,503 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:50,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:50,505 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:50,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:50,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:50,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:50,505 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:50,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:50,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:50,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,512 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:50,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:50,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:50,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:50,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:50,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:50,519 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:50,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:50,519 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:50,519 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:50,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,520 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326850592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326850593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326850593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326850596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326850596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,620 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:50,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:50,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:50,632 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:50,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:50,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:50,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:50,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,635 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,635 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:50,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:50,635 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,636 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:50,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:50,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:50,640 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,640 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:50,640 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:50,640 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:50,640 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:50,644 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,644 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,644 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,644 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:50,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41727db5[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:50,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:50,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,652 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,652 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:50,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:50,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:50,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:50,660 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:50,660 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,660 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:50,661 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:50,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:50,662 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:50,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:50,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:50,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:50,662 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:50,669 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,669 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,669 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,669 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,670 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:50,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:50,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:50,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:50,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:50,677 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:50,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:50,678 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,678 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:50,678 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,678 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:50,678 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,678 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,778 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:50,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:50,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:50,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:50,781 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:50,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:50,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:50,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:50,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,785 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:50,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:50,785 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,786 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,786 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:50,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:50,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:50,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:50,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:50,786 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:50,787 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,787 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:50,787 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:50,787 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:50,787 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:50,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326850798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326850798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326850798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326850799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:50,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326850799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,813 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,813 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,813 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:50,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72211335[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:50,814 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:50,814 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,827 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,827 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:50,827 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:50,827 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:50,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,827 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:50,835 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:50,835 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,835 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,835 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:50,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:50,837 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:50,837 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:50,837 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:50,837 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:50,837 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:50,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:50,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,844 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,844 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,847 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:50,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:50,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:50,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,852 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:50,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,853 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,852 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:50,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:50,852 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:50,853 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:50,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:50,890 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042407d010f3005e41079aaf55a6064d0d94 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042407d010f3005e41079aaf55a6064d0d94 2023-04-24 08:59:50,896 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/3fcc0b508b1646f09533d4b48b6641bc 2023-04-24 08:59:50,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/5dd2a512fb92405b9f3aa3bc393906c3 2023-04-24 08:59:50,953 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:50,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:50,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:50,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:50,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:50,956 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:50,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:50,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:50,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:50,958 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,958 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,968 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:50,968 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:50,968 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,968 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:50,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:50,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:50,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:50,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:50,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:50,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:50,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:50,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:50,969 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,969 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:50,970 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:50,970 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:50,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:50,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:50,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:50,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:50,977 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,977 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,977 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,977 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:50,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525ac0e3[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:50,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:50,979 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,985 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:50,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:50,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:50,986 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,986 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:50,986 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:50,994 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:50,994 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,994 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:50,994 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:50,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:50,996 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:50,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:50,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:50,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:50,996 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:51,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:51,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:51,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,010 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:51,010 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:51,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:51,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:51,010 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,010 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:51,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:51,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:51,011 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:51,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,011 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:51,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,011 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:51,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326851102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326851103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326851104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326851105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326851105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,111 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:51,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:51,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:51,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:51,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:51,114 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:51,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:51,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:51,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:51,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,119 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:51,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:51,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,119 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:51,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:51,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:51,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:51,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:51,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:51,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:51,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:51,120 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:51,120 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:51,120 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:51,120 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:51,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:51,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:51,120 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,121 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:51,121 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:51,127 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,127 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:51,127 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,127 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:51,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:51,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:51,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ee4d8a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:51,129 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,135 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:51,135 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,136 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:51,135 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:51,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:51,144 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:51,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:51,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:51,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:51,146 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:51,146 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:51,146 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:51,146 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:51,146 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:51,152 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,152 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:51,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,152 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:51,153 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,153 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,160 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:51,160 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:51,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:51,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:51,160 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:51,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:51,160 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:51,160 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,161 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:51,161 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:51,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:51,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,161 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,261 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:51,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:51,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:51,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:51,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:51,264 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:51,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:51,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:51,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:51,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,269 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,269 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:51,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:51,269 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,269 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:51,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:51,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:51,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:51,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:51,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:51,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:51,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:51,270 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:51,270 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:51,270 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:51,270 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:51,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:51,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:51,270 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,271 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:51,271 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:51,277 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,277 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:51,277 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,277 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:51,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:51,278 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:51,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33230bd7[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:51,278 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,285 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,286 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:51,285 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:51,286 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:51,286 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:51,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:51,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,314 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:51,314 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:51,315 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:51,316 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:51,316 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:51,316 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:51,316 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:51,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/da591db4a47d4fc8b85d898f6d7c4b0d 2023-04-24 08:59:51,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/3fcc0b508b1646f09533d4b48b6641bc as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/3fcc0b508b1646f09533d4b48b6641bc 2023-04-24 08:59:51,327 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,327 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:51,327 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:51,328 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,329 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/3fcc0b508b1646f09533d4b48b6641bc, entries=150, sequenceid=16, filesize=25.1 K 2023-04-24 08:59:51,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/5dd2a512fb92405b9f3aa3bc393906c3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/5dd2a512fb92405b9f3aa3bc393906c3 2023-04-24 08:59:51,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/5dd2a512fb92405b9f3aa3bc393906c3, entries=150, sequenceid=16, filesize=11.6 K 2023-04-24 08:59:51,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/da591db4a47d4fc8b85d898f6d7c4b0d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/da591db4a47d4fc8b85d898f6d7c4b0d 2023-04-24 08:59:51,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:51,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:51,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:51,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:51,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:51,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:51,335 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:51,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:51,336 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:51,336 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:51,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:51,336 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/da591db4a47d4fc8b85d898f6d7c4b0d, entries=150, sequenceid=16, filesize=11.6 K 2023-04-24 08:59:51,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d7e0c211118009ab258bf68c7108cae8 in 864ms, sequenceid=16, compaction requested=false 2023-04-24 08:59:51,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:51,436 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:51,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:51,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:51,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:51,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:51,439 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:51,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:51,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:51,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:51,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,444 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,444 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:51,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:51,444 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,444 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:51,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:51,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:51,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:51,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:51,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:51,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:51,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:51,445 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:51,446 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:51,446 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:51,446 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 08:59:51,446 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 08:59:51,446 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:51,446 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 08:59:51,446 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:51,446 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 08:59:51,446 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:51,457 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304241e47e3bfad894231a70ffe4404f6ec9b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304241e47e3bfad894231a70ffe4404f6ec9b 2023-04-24 08:59:51,462 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/c8f5cb9622854b6ab90fc5c1e65b4f0c 2023-04-24 08:59:51,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:51,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:51,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326851615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326851615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326851616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326851616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326851617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326851718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326851720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326851721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326851721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326851721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,871 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/f9fa879f43dc4846a2569c6264f92cb9 2023-04-24 08:59:51,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326851922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326851923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326851923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326851924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:51,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:51,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326851924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,066 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 08:59:52,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326852226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326852226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326852227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326852227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326852228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,280 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/11a5a8789b124f3ca0c1a5ad51c74ff0 2023-04-24 08:59:52,283 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/c8f5cb9622854b6ab90fc5c1e65b4f0c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c8f5cb9622854b6ab90fc5c1e65b4f0c 2023-04-24 08:59:52,286 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c8f5cb9622854b6ab90fc5c1e65b4f0c, entries=150, sequenceid=36, filesize=25.1 K 2023-04-24 08:59:52,287 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/f9fa879f43dc4846a2569c6264f92cb9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f9fa879f43dc4846a2569c6264f92cb9 2023-04-24 08:59:52,290 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f9fa879f43dc4846a2569c6264f92cb9, entries=150, sequenceid=36, filesize=11.6 K 2023-04-24 08:59:52,291 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/11a5a8789b124f3ca0c1a5ad51c74ff0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/11a5a8789b124f3ca0c1a5ad51c74ff0 2023-04-24 08:59:52,298 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/11a5a8789b124f3ca0c1a5ad51c74ff0, entries=150, sequenceid=36, filesize=11.6 K 2023-04-24 08:59:52,299 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d7e0c211118009ab258bf68c7108cae8 in 853ms, sequenceid=36, compaction requested=false 2023-04-24 08:59:52,299 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:52,299 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:52,299 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:52,299 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:52,299 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,299 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:52,300 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:52,310 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:52,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:52,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:52,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:52,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49610e3d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:52,311 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:52,311 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:52,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:52,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:52,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:52,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:52,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:52,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:52,327 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,327 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:52,327 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:52,327 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:52,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:52,329 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:52,329 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:52,329 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:52,329 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:52,329 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:52,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:52,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:52,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:52,336 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:52,337 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:52,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:52,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:52,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:52,344 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:52,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:52,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:52,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:52,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:52,345 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:52,345 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:52,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:52,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:52,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:52,345 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,445 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:52,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:52,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:52,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:52,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:52,447 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:52,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:52,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:52,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:52,448 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,448 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:52,452 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:52,452 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:52,452 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,452 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:52,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:52,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:52,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:52,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:52,453 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:52,453 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:52,453 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:52,453 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:52,453 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:52,453 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:52,453 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:52,454 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 08:59:52,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 08:59:52,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:52,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 08:59:52,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:52,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 08:59:52,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:52,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:52,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:52,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326852752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326852755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326852755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326852756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326852757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326852861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326852862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326852863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326852863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326852863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:52,879 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304243a5314084a084fb4b10e6acbeb879509 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243a5314084a084fb4b10e6acbeb879509 2023-04-24 08:59:53,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326853065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326853068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326853068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326853068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326853069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,284 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/499c6189073f4812aacb7d50dc7bf790 2023-04-24 08:59:53,292 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/a957da97655f47e4a24accb702e0a2e9 2023-04-24 08:59:53,304 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/78b8de05792c42fdb085d619ed4c4217 2023-04-24 08:59:53,307 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/499c6189073f4812aacb7d50dc7bf790 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/499c6189073f4812aacb7d50dc7bf790 2023-04-24 08:59:53,310 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/499c6189073f4812aacb7d50dc7bf790, entries=150, sequenceid=48, filesize=25.1 K 2023-04-24 08:59:53,311 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/a957da97655f47e4a24accb702e0a2e9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a957da97655f47e4a24accb702e0a2e9 2023-04-24 08:59:53,315 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a957da97655f47e4a24accb702e0a2e9, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 08:59:53,316 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/78b8de05792c42fdb085d619ed4c4217 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/78b8de05792c42fdb085d619ed4c4217 2023-04-24 08:59:53,318 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/78b8de05792c42fdb085d619ed4c4217, entries=150, sequenceid=48, filesize=11.6 K 2023-04-24 08:59:53,319 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d7e0c211118009ab258bf68c7108cae8 in 865ms, sequenceid=48, compaction requested=true 2023-04-24 08:59:53,319 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:53,319 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:53,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:53,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:53,319 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:53,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:53,352 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,353 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:53,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d38922a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:53,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:53,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,360 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:53,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:53,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:53,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:53,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:53,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:53,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:53,371 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:53,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:53,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:53,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:53,371 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:53,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:53,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-24 08:59:53,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 08:59:53,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:53,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 08:59:53,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:53,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 08:59:53,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:53,377 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,378 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,378 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,386 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:53,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:53,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:53,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:53,386 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:53,386 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:53,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:53,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:53,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:53,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:53,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326853381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326853383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326853384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326853388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326853389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,486 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:53,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:53,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:53,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:53,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:53,489 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:53,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:53,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:53,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:53,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,490 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:53,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:53,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:53,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:53,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:53,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:53,495 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:53,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326853489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,495 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:53,495 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:53,495 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:53,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:53,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326853491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:53,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:53,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326853491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326853491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326853497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,502 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:53,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea9141a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:53,503 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:53,503 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,510 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:53,510 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:53,510 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:53,510 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:53,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,519 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,519 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:53,519 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:53,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:53,520 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:53,521 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:53,521 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:53,521 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:53,521 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:53,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:53,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,549 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,568 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:53,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,568 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:53,568 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:53,569 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:53,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:53,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:53,669 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:53,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:53,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:53,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:53,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:53,671 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:53,671 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:53,671 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:53,671 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:53,672 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,672 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,677 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:53,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:53,677 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,677 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:53,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:53,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:53,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:53,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:53,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:53,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:53,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:53,678 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:53,678 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:53,679 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:53,679 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:53,679 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:53,679 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:53,679 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:53,679 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,679 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:53,679 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:53,685 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,685 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,686 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,686 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:53,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75ffc1ab[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:53,687 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:53,687 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,694 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,694 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:53,694 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:53,694 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,694 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:53,694 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:53,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326853696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326853696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326853697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,702 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:53,702 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:53,702 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:53,702 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326853697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:53,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:53,704 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:53,704 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:53,704 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:53,704 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:53,704 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:53,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:53,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326853703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,710 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,710 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:53,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,710 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:53,711 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,712 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:53,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,719 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:53,719 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,719 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:53,719 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:53,788 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042458bb6680fc1b4ff4b881dac419135567 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042458bb6680fc1b4ff4b881dac419135567 2023-04-24 08:59:53,819 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:53,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:53,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:53,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:53,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:53,822 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:53,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:53,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:53,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:53,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,978 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:53,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:53,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:53,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:53,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:53,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:53,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:53,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:53,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:53,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:53,981 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:53,981 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:53,981 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:53,981 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:53,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:53,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:53,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:53,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:53,993 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,993 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:53,993 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:53,993 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:53,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:53,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@342e29cf[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:53,994 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:53,994 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,002 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:54,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:54,002 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:54,002 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:54,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326854000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326854001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326854002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326854005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,010 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,010 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,010 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,011 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326854009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:54,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:54,013 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:54,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:54,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:54,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:54,013 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:54,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,020 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,027 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:54,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:54,027 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:54,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,027 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:54,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,028 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,028 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,127 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:54,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:54,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:54,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:54,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:54,130 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:54,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:54,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:54,131 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,131 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,144 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,144 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,144 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,144 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:54,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:54,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:54,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:54,145 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,145 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:54,145 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:54,145 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:54,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:54,152 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:54,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d8e44a2[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:54,154 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:54,154 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,160 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,160 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:54,161 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:54,161 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,161 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:54,161 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:54,169 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:54,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,169 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,169 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:54,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:54,170 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:54,170 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:54,170 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:54,170 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:54,170 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:54,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,177 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,192 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/ba93366242b24b7da5b9b5c086ab9423 2023-04-24 08:59:54,219 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,219 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,220 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:54,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:54,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:54,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,221 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:54,221 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:54,321 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:54,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:54,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:54,323 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:54,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:54,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:54,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,324 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,327 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,327 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,327 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,327 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:54,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:54,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:54,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:54,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:54,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:54,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:54,328 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,329 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:54,329 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:54,329 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:54,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:54,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:54,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:54,335 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:54,337 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:54,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b118ecd[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:54,337 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,344 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:54,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:54,344 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,344 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:54,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:54,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:54,352 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:54,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:54,354 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:54,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:54,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:54,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:54,354 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:54,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:54,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,369 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,369 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:54,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:54,369 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:54,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,469 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:54,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:54,472 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:54,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:54,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:54,473 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,473 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,477 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,477 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,477 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,477 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:54,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:54,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:54,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:54,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:54,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:54,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:54,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:54,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:54,480 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,480 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:54,480 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:54,480 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,480 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:54,480 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:54,485 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,485 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,486 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:54,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20e8f09d[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:54,487 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:54,487 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:54,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:54,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:54,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:54,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:54,502 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:54,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:54,504 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:54,504 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:54,504 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:54,504 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:54,504 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:54,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326854506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326854506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326854508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,519 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:54,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:54,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,519 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,519 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326854509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:54,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326854516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/7be79b168dbc41279987b0f9599ce9b3 2023-04-24 08:59:54,619 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:54,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:54,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:54,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:54,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:54,623 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:54,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:54,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:54,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,627 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,627 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,627 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,627 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:54,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:54,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:54,628 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,628 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:54,628 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:54,628 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:54,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:54,635 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,635 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,635 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,636 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:54,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cdd51e[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:54,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:54,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,644 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,644 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:54,644 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,644 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:54,644 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:54,644 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:54,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:54,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,652 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:54,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:54,654 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:54,654 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:54,654 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:54,654 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:54,654 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:54,660 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,660 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,660 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,661 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,687 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,687 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,687 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:54,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:54,687 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,687 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,687 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:54,687 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,687 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:54,687 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,788 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:54,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:54,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:54,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:54,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:54,790 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:54,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:54,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:54,791 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,791 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,794 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,794 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,794 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,794 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:54,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:54,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:54,795 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,795 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:54,795 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:54,795 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:54,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:54,802 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:54,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a293651[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:54,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:54,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,810 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:54,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:54,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:54,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,811 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:54,819 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,819 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:54,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:54,821 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:54,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:54,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:54,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:54,821 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:54,827 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,827 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,827 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,828 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,829 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,835 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,835 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,835 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,835 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:54,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:54,835 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:54,835 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,836 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,836 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:54,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,936 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:54,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:54,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:54,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:54,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:54,938 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:54,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:54,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:54,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,944 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,944 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,944 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,944 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:54,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:54,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:54,945 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,945 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:54,945 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:54,945 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:54,945 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:54,952 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,952 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:54,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d69a23f[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:54,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:54,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,960 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:54,961 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:54,961 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:54,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:54,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:54,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,969 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:54,969 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:54,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:54,971 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:54,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:54,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:54,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:54,971 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:54,977 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,977 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:54,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,977 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,977 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:54,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,985 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:54,985 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:54,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:54,985 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,986 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:54,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:54,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:54,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:54,986 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:54,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:54,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:54,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:54,986 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:54,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:54,986 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:55,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/1128bf44fbfc41dba5cf6bc708a18eeb 2023-04-24 08:59:55,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/ba93366242b24b7da5b9b5c086ab9423 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/ba93366242b24b7da5b9b5c086ab9423 2023-04-24 08:59:55,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/ba93366242b24b7da5b9b5c086ab9423, entries=200, sequenceid=69, filesize=31.9 K 2023-04-24 08:59:55,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/7be79b168dbc41279987b0f9599ce9b3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/7be79b168dbc41279987b0f9599ce9b3 2023-04-24 08:59:55,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/7be79b168dbc41279987b0f9599ce9b3, entries=150, sequenceid=69, filesize=11.6 K 2023-04-24 08:59:55,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/1128bf44fbfc41dba5cf6bc708a18eeb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/1128bf44fbfc41dba5cf6bc708a18eeb 2023-04-24 08:59:55,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/1128bf44fbfc41dba5cf6bc708a18eeb, entries=150, sequenceid=69, filesize=11.6 K 2023-04-24 08:59:55,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for d7e0c211118009ab258bf68c7108cae8 in 1651ms, sequenceid=69, compaction requested=true 2023-04-24 08:59:55,027 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:55,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:55,027 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:55,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:55,027 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:55,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:55,028 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:55,028 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:55,028 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/A is initiating minor compaction (all files) 2023-04-24 08:59:55,028 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/B is initiating minor compaction (all files) 2023-04-24 08:59:55,028 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/A in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:55,028 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/B in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:55,029 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/3fcc0b508b1646f09533d4b48b6641bc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c8f5cb9622854b6ab90fc5c1e65b4f0c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/499c6189073f4812aacb7d50dc7bf790, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/ba93366242b24b7da5b9b5c086ab9423] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=107.3 K 2023-04-24 08:59:55,029 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/5dd2a512fb92405b9f3aa3bc393906c3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f9fa879f43dc4846a2569c6264f92cb9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a957da97655f47e4a24accb702e0a2e9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/7be79b168dbc41279987b0f9599ce9b3] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=46.4 K 2023-04-24 08:59:55,029 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 3fcc0b508b1646f09533d4b48b6641bc, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682326790467 2023-04-24 08:59:55,029 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5dd2a512fb92405b9f3aa3bc393906c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682326790467 2023-04-24 08:59:55,029 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c8f5cb9622854b6ab90fc5c1e65b4f0c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682326790486 2023-04-24 08:59:55,029 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f9fa879f43dc4846a2569c6264f92cb9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682326790486 2023-04-24 08:59:55,029 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 499c6189073f4812aacb7d50dc7bf790, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682326791615 2023-04-24 08:59:55,030 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a957da97655f47e4a24accb702e0a2e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682326791615 2023-04-24 08:59:55,030 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting ba93366242b24b7da5b9b5c086ab9423, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682326792755 2023-04-24 08:59:55,030 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7be79b168dbc41279987b0f9599ce9b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682326792755 2023-04-24 08:59:55,048 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#B#compaction#270 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:55,060 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#A#compaction#271 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:55,064 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/996046a403804cf99a6eacc7a19861a7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/996046a403804cf99a6eacc7a19861a7 2023-04-24 08:59:55,070 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/B of d7e0c211118009ab258bf68c7108cae8 into 996046a403804cf99a6eacc7a19861a7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:55,070 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:55,070 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/B, priority=12, startTime=1682326795027; duration=0sec 2023-04-24 08:59:55,070 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:55,070 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:55,071 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:55,071 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/C is initiating minor compaction (all files) 2023-04-24 08:59:55,071 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/C in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:55,071 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/da591db4a47d4fc8b85d898f6d7c4b0d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/11a5a8789b124f3ca0c1a5ad51c74ff0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/78b8de05792c42fdb085d619ed4c4217, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/1128bf44fbfc41dba5cf6bc708a18eeb] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=46.4 K 2023-04-24 08:59:55,072 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting da591db4a47d4fc8b85d898f6d7c4b0d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682326790467 2023-04-24 08:59:55,072 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 11a5a8789b124f3ca0c1a5ad51c74ff0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682326790486 2023-04-24 08:59:55,075 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 78b8de05792c42fdb085d619ed4c4217, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682326791615 2023-04-24 08:59:55,076 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 1128bf44fbfc41dba5cf6bc708a18eeb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682326792755 2023-04-24 08:59:55,086 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:55,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:55,087 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#C#compaction#272 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:55,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:55,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:55,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:55,089 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:55,089 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:55,089 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:55,089 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:55,089 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:55,089 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:55,094 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,094 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:55,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:55,094 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,094 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:55,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:55,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:55,097 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/a69a130e5273421eaf51ad28b13cbe17 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a69a130e5273421eaf51ad28b13cbe17 2023-04-24 08:59:55,102 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/C of d7e0c211118009ab258bf68c7108cae8 into a69a130e5273421eaf51ad28b13cbe17(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:55,102 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:55,102 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/C, priority=12, startTime=1682326795027; duration=0sec 2023-04-24 08:59:55,102 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:55,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:55,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:55,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:55,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:55,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:55,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:55,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:55,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:55,104 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 08:59:55,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 08:59:55,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:55,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 08:59:55,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:55,104 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 08:59:55,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:55,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:55,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:55,527 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424532358a2ebcc496789159fab863ae7ed to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424532358a2ebcc496789159fab863ae7ed 2023-04-24 08:59:55,532 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/1cebbf014be4488aac7726d4a7408814 2023-04-24 08:59:55,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326855541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326855543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326855543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682326795529, max timestamp=1682326795549Num uniques 859; 2023-04-24 08:59:55,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682326795529, max timestamp=1682326795549Num uniques 859; 2023-04-24 08:59:55,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-24 08:59:55,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682326795529, max timestamp=1682326795549Num uniques 859; 2023-04-24 08:59:55,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-24 08:59:55,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-24 08:59:55,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326855593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326855599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326855652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326855653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326855653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326855700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326855705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326855858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326855858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326855858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,895 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/e7a7d9894b1e4a5fbfd2a041e79203b0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e7a7d9894b1e4a5fbfd2a041e79203b0 2023-04-24 08:59:55,898 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/A of d7e0c211118009ab258bf68c7108cae8 into e7a7d9894b1e4a5fbfd2a041e79203b0(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:55,898 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:55,898 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/A, priority=12, startTime=1682326795027; duration=0sec 2023-04-24 08:59:55,898 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:55,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326855906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:55,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326855907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:55,940 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/cded3c10c5cc441c90411d71ddca38e6 2023-04-24 08:59:56,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326856161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326856163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326856164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326856213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326856213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,349 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/0cd858b2ed124ace870e50a41fa554aa 2023-04-24 08:59:56,352 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/1cebbf014be4488aac7726d4a7408814 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1cebbf014be4488aac7726d4a7408814 2023-04-24 08:59:56,355 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1cebbf014be4488aac7726d4a7408814, entries=150, sequenceid=82, filesize=25.1 K 2023-04-24 08:59:56,356 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/cded3c10c5cc441c90411d71ddca38e6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/cded3c10c5cc441c90411d71ddca38e6 2023-04-24 08:59:56,359 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/cded3c10c5cc441c90411d71ddca38e6, entries=150, sequenceid=82, filesize=11.6 K 2023-04-24 08:59:56,359 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/0cd858b2ed124ace870e50a41fa554aa as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/0cd858b2ed124ace870e50a41fa554aa 2023-04-24 08:59:56,362 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/0cd858b2ed124ace870e50a41fa554aa, entries=150, sequenceid=82, filesize=11.6 K 2023-04-24 08:59:56,363 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for d7e0c211118009ab258bf68c7108cae8 in 1259ms, sequenceid=82, compaction requested=false 2023-04-24 08:59:56,363 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:56,363 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:56,363 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:56,363 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:56,363 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,367 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:56,367 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:56,402 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,402 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:56,402 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:56,402 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:56,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:56,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae2c717[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:56,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:56,403 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:56,410 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:56,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:56,410 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:56,410 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:56,410 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:56,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,411 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:56,419 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:56,419 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:56,419 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,419 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:56,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:56,421 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:56,421 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:56,421 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:56,421 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:56,421 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:56,427 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,427 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,427 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:56,427 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:56,428 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:56,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:56,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:56,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:56,436 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:56,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:56,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:56,435 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:56,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:56,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:56,436 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:56,436 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:56,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:56,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:56,436 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,536 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:56,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:56,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:56,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:56,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:56,539 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:56,539 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:56,539 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:56,539 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:56,539 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,539 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:56,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,544 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:56,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:56,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:56,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:56,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:56,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:56,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:56,545 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:56,545 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:56,545 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:56,545 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:56,545 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:56,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:56,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:56,546 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-24 08:59:56,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 08:59:56,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:56,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 08:59:56,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:56,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 08:59:56,546 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 08:59:56,556 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042448b45d7afdbd4ece965ad21800f921f2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042448b45d7afdbd4ece965ad21800f921f2 2023-04-24 08:59:56,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:56,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:56,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326856677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326856680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326856681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326856718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326856721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326856782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326856783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326856786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,960 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=116, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/9c925dfc7b004f4f9608182e34438852 2023-04-24 08:59:56,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326856985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326856990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:56,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:56,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326856991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:57,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326857293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:57,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326857297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:57,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326857297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,375 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/feea3472a815434587ff1300085fec11 2023-04-24 08:59:57,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:57,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326857728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:57,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326857732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,784 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/372e0a8ae1ab4dd595212eae5d3a70de 2023-04-24 08:59:57,787 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/9c925dfc7b004f4f9608182e34438852 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/9c925dfc7b004f4f9608182e34438852 2023-04-24 08:59:57,790 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/9c925dfc7b004f4f9608182e34438852, entries=150, sequenceid=116, filesize=25.1 K 2023-04-24 08:59:57,791 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/feea3472a815434587ff1300085fec11 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/feea3472a815434587ff1300085fec11 2023-04-24 08:59:57,794 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/feea3472a815434587ff1300085fec11, entries=150, sequenceid=116, filesize=11.6 K 2023-04-24 08:59:57,794 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/372e0a8ae1ab4dd595212eae5d3a70de as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/372e0a8ae1ab4dd595212eae5d3a70de 2023-04-24 08:59:57,797 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/372e0a8ae1ab4dd595212eae5d3a70de, entries=150, sequenceid=116, filesize=11.6 K 2023-04-24 08:59:57,798 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for d7e0c211118009ab258bf68c7108cae8 in 1252ms, sequenceid=116, compaction requested=true 2023-04-24 08:59:57,798 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:57,798 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:57,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:57,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:57,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:57,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:57,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:57,832 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 08:59:57,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 08:59:57,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:57,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 08:59:57,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:57,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 08:59:57,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:57,872 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,872 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:57,872 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:57,872 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:57,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:57,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@610c8a70[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:57,874 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:57,875 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:57,885 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:57,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:57,885 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:57,885 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,885 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:57,885 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:57,886 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:57,894 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:57,894 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,894 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:57,894 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:57,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:57,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:57,897 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:57,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:57,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:57,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:57,897 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:57,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:57,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326857895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:57,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326857896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,902 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:57,902 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:57,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:57,902 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:57,902 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:57,903 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326857897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,910 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,910 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:57,910 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:57,910 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:57,910 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:57,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:57,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:57,910 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:57,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:57,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:57,911 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:57,910 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:57,911 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:57,911 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:57,911 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:57,911 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:57,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326858001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326858002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,011 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:58,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:58,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326858007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:58,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:58,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:58,014 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:58,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:58,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:58,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:58,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:58,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:58,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:58,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:58,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:58,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:58,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:58,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:58,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:58,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:58,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,021 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:58,021 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:58,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,027 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,027 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,027 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:58,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274b3be[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:58,028 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:58,028 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,035 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,035 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,035 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:58,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:58,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:58,036 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:58,043 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:58,044 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,043 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,044 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:58,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:58,045 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:58,046 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:58,046 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:58,046 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:58,046 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:58,052 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,052 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,052 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,052 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,053 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,060 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,060 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,060 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:58,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:58,060 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,061 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:58,060 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,061 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:58,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,061 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:58,061 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,061 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,061 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,161 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:58,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:58,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:58,164 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:58,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:58,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:58,165 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,165 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,169 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,169 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,170 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:58,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:58,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:58,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:58,170 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:58,170 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,170 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:58,170 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:58,170 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:58,170 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,170 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:58,171 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:58,171 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,171 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:58,171 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:58,171 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,171 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:58,171 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:58,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:58,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42cc9195[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:58,178 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:58,179 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,185 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:58,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:58,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:58,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:58,193 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,193 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,194 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,194 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:58,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:58,195 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:58,195 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:58,195 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:58,195 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:58,195 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:58,202 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,202 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,202 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,203 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,210 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:58,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:58,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:58,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,211 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:58,211 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:58,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326858206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326858207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326858212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,247 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304243321957880194c32833b1e02d8fdcedd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243321957880194c32833b1e02d8fdcedd 2023-04-24 08:59:58,311 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:58,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:58,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:58,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:58,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:58,314 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:58,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:58,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:58,315 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,315 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:58,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:58,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:58,320 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,320 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:58,320 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:58,320 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:58,320 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:58,327 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,327 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,327 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,327 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:58,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6a2ca4[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:58,328 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:58,328 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:58,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:58,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:58,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:58,368 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:58,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:58,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:58,371 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:58,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:58,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:58,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:58,371 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:58,377 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,377 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,377 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,377 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,378 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,385 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:58,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:58,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,386 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:58,386 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:58,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:58,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,486 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:58,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:58,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:58,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:58,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:58,488 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:58,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:58,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:58,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:58,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:58,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:58,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:58,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:58,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:58,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:58,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:58,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:58,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:58,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:58,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:58,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:58,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,502 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:58,504 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:58,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb06e13[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:58,504 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,510 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:58,510 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:58,510 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,510 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:58,510 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:58,519 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:58,519 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,519 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:58,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326858516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326858517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:58,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326858517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:58,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:58,521 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:58,521 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:58,521 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:58,521 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:58,521 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:58,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,528 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,535 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,535 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:58,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:58,535 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:58,536 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:58,536 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:58,535 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,535 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,536 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,636 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:58,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:58,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:58,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:58,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:58,639 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:58,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:58,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:58,640 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,640 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,644 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,644 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,644 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,644 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:58,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:58,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:58,645 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,645 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:58,645 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:58,645 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:58,645 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:58,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,652 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,652 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:58,653 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:58,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cfec398[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:58,653 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,655 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=128, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/11563e8f6057410098ae7f83d9d06ae3 2023-04-24 08:59:58,661 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,661 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:58,661 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:58,661 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:58,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,662 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:58,669 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:58,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:58,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:58,671 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:58,671 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:58,671 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:58,671 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:58,671 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:58,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,677 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,678 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,678 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,685 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,685 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,685 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,685 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:58,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,685 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:58,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:58,685 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,686 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:58,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,686 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:58,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,786 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:58,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:58,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:58,789 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:58,789 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,789 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:58,789 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:58,789 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,789 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,794 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,794 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,794 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,794 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:58,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:58,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:58,795 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,795 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:58,795 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:58,795 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:58,795 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:58,802 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:58,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a18a9ab[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:58,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:58,803 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,810 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,810 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,810 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:58,810 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:58,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:58,811 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:58,836 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:58,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,836 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,836 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:58,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:58,838 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:58,838 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:58,838 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:58,838 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:58,838 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:58,851 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,851 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,851 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,852 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,852 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,860 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,860 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,860 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:58,860 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,860 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:58,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:58,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:58,860 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:58,861 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,861 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:58,861 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:58,861 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:58,861 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,861 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:58,961 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:58,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:58,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:58,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:58,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:58,964 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:58,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:58,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:58,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,977 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:58,977 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:58,977 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,977 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:58,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:58,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:58,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:58,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:58,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:58,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:58,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:58,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:58,978 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:58,978 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,978 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:58,979 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:58,979 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:58,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:58,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:58,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:58,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:58,993 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,993 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:58,993 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:58,993 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:58,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:58,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55f8ad40[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:58,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:58,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,002 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:59,002 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:59,002 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:59,002 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:59,010 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,010 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:59,010 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,010 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:59,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:59,012 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:59,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:59,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:59,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:59,012 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:59,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,019 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,020 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:59,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326859021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:59,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326859021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:59,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326859022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:59,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:59,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:59,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:59,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:59,027 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:59,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:59,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,027 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:59,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:59,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,027 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:59,027 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:59,028 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/55557db42b534af3a84e1f2066c55168 2023-04-24 08:59:59,128 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:59,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:59,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:59,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:59,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:59,130 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:59,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:59,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:59,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:59,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,135 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:59,135 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:59,136 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,136 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:59,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:59,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:59,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:59,137 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,137 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:59,137 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:59,137 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:59,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:59,144 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:59,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d80811a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:59,145 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:59,145 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,152 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:59,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:59,152 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:59,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,152 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:59,160 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:59,160 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,160 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,160 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:59,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:59,163 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:59,163 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:59,163 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:59,163 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:59,163 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:59,168 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,169 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:59,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,169 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,170 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:59,177 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:59,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:59,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:59,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:59,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:59,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:59,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:59,177 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:59,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:59,177 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:59,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,178 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,178 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,178 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,178 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,278 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:59,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:59,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:59,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:59,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:59,281 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:59,281 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:59,281 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:59,281 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:59,281 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,281 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,285 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,285 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:59,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:59,286 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,286 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:59,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:59,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:59,287 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,287 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:59,287 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:59,287 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:59,287 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:59,294 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,294 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,294 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,294 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:59,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7086d31[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:59,295 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:59,295 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,302 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,302 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:59,302 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:59,302 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,302 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:59,302 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:59,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:59,310 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,310 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:59,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:59,312 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:59,312 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:59,312 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:59,312 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:59,312 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:59,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,319 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:59,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:59,319 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,320 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:59,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:59,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:59,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:59,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:59,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:59,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:59,361 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:59,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,361 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:59,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:59,361 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,362 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,362 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,461 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:59,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:59,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:59,469 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:59,469 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:59,470 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:59,470 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:59,470 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,470 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/170fa559d7584fce847edce39151c06d 2023-04-24 08:59:59,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/11563e8f6057410098ae7f83d9d06ae3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/11563e8f6057410098ae7f83d9d06ae3 2023-04-24 08:59:59,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/11563e8f6057410098ae7f83d9d06ae3, entries=150, sequenceid=128, filesize=25.1 K 2023-04-24 08:59:59,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/55557db42b534af3a84e1f2066c55168 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/55557db42b534af3a84e1f2066c55168 2023-04-24 08:59:59,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/55557db42b534af3a84e1f2066c55168, entries=100, sequenceid=128, filesize=9.3 K 2023-04-24 08:59:59,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/170fa559d7584fce847edce39151c06d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/170fa559d7584fce847edce39151c06d 2023-04-24 08:59:59,485 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:59,485 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:59,486 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,486 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:59,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:59,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:59,487 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,487 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:59,487 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:59,487 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 08:59:59,487 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 08:59:59,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/170fa559d7584fce847edce39151c06d, entries=100, sequenceid=128, filesize=9.3 K 2023-04-24 08:59:59,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d7e0c211118009ab258bf68c7108cae8 in 1657ms, sequenceid=128, compaction requested=true 2023-04-24 08:59:59,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:59,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:59,489 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:59,489 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:59,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:59,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:59,490 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103118 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:59,490 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/A is initiating minor compaction (all files) 2023-04-24 08:59:59,490 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/A in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,490 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45338 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:59,490 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e7a7d9894b1e4a5fbfd2a041e79203b0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1cebbf014be4488aac7726d4a7408814, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/9c925dfc7b004f4f9608182e34438852, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/11563e8f6057410098ae7f83d9d06ae3] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=100.7 K 2023-04-24 08:59:59,490 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/B is initiating minor compaction (all files) 2023-04-24 08:59:59,490 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/B in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,490 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/996046a403804cf99a6eacc7a19861a7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/cded3c10c5cc441c90411d71ddca38e6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/feea3472a815434587ff1300085fec11, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/55557db42b534af3a84e1f2066c55168] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=44.3 K 2023-04-24 08:59:59,491 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 996046a403804cf99a6eacc7a19861a7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682326792755 2023-04-24 08:59:59,491 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e7a7d9894b1e4a5fbfd2a041e79203b0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682326792755 2023-04-24 08:59:59,491 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting cded3c10c5cc441c90411d71ddca38e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326793379 2023-04-24 08:59:59,491 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1cebbf014be4488aac7726d4a7408814, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326793379 2023-04-24 08:59:59,491 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting feea3472a815434587ff1300085fec11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682326795580 2023-04-24 08:59:59,491 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 9c925dfc7b004f4f9608182e34438852, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682326795580 2023-04-24 08:59:59,492 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 55557db42b534af3a84e1f2066c55168, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326796679 2023-04-24 08:59:59,492 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 11563e8f6057410098ae7f83d9d06ae3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326796679 2023-04-24 08:59:59,494 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,494 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 08:59:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 08:59:59,496 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 08:59:59,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78a2c98a[Count = 0] remaining members to acquire global barrier 2023-04-24 08:59:59,496 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,499 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#B#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:59,502 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 08:59:59,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 08:59:59,502 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 08:59:59,502 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,503 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 08:59:59,503 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#A#compaction#283 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:59,507 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/312ccc8776f04a55aeb90a2d840e317b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/312ccc8776f04a55aeb90a2d840e317b 2023-04-24 08:59:59,510 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,511 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 08:59:59,511 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:59,511 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 08:59:59,511 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/B of d7e0c211118009ab258bf68c7108cae8 into 312ccc8776f04a55aeb90a2d840e317b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:59,511 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:59,511 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/B, priority=12, startTime=1682326799489; duration=0sec 2023-04-24 08:59:59,511 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 08:59:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,512 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 08:59:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,513 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45338 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 08:59:59,513 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/C is initiating minor compaction (all files) 2023-04-24 08:59:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,513 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/C in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,513 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a69a130e5273421eaf51ad28b13cbe17, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/0cd858b2ed124ace870e50a41fa554aa, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/372e0a8ae1ab4dd595212eae5d3a70de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/170fa559d7584fce847edce39151c06d] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=44.3 K 2023-04-24 08:59:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,513 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a69a130e5273421eaf51ad28b13cbe17, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682326792755 2023-04-24 08:59:59,514 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 0cd858b2ed124ace870e50a41fa554aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326793379 2023-04-24 08:59:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 08:59:59,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 08:59:59,514 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 08:59:59,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 08:59:59,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 08:59:59,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 08:59:59,514 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 08:59:59,515 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 372e0a8ae1ab4dd595212eae5d3a70de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682326795580 2023-04-24 08:59:59,515 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 170fa559d7584fce847edce39151c06d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326796679 2023-04-24 08:59:59,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:59,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,519 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 08:59:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 08:59:59,519 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 08:59:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,520 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#C#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 08:59:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 08:59:59,521 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 08:59:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 08:59:59,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:59,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 08:59:59,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:59,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 08:59:59,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 08:59:59,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 08:59:59,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 08:59:59,527 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 08:59:59,528 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,528 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 08:59:59,528 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 08:59:59,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 08:59:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 08:59:59,628 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 08:59:59,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 08:59:59,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 08:59:59,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 08:59:59,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 08:59:59,631 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 08:59:59,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:59,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 08:59:59,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 08:59:59,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,635 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 08:59:59,635 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 08:59:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 08:59:59,636 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,636 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 08:59:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 08:59:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 08:59:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 08:59:59,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 08:59:59,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 08:59:59,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 08:59:59,637 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 08:59:59,637 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 08:59:59,637 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 08:59:59,637 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 08:59:59,637 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 08:59:59,637 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 08:59:59,637 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 08:59:59,637 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 08:59:59,637 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 08:59:59,637 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:59,637 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 08:59:59,637 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:59,637 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 08:59:59,637 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 08:59:59,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 08:59:59,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 08:59:59,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:59,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326859753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:59,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326859755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:59,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326859857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 08:59:59,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326859860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 08:59:59,928 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/540671d3bfe347cdb676c4872007fdd3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/540671d3bfe347cdb676c4872007fdd3 2023-04-24 08:59:59,932 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/C of d7e0c211118009ab258bf68c7108cae8 into 540671d3bfe347cdb676c4872007fdd3(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:59,932 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/cf211ad810d14cb9bd088049ca6b4415 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cf211ad810d14cb9bd088049ca6b4415 2023-04-24 08:59:59,932 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:59,932 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/C, priority=12, startTime=1682326799489; duration=0sec 2023-04-24 08:59:59,932 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 08:59:59,936 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/A of d7e0c211118009ab258bf68c7108cae8 into cf211ad810d14cb9bd088049ca6b4415(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 08:59:59,936 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 08:59:59,936 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/A, priority=12, startTime=1682326799489; duration=0sec 2023-04-24 08:59:59,936 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:00,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326860026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:00,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326860032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:00,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326860035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:00,049 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424e4f3f4cb8e774ed5b04fa0d5853f1c9c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e4f3f4cb8e774ed5b04fa0d5853f1c9c 2023-04-24 09:00:00,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326860061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:00,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326860065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:00,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326860365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:00,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326860370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:00,459 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/a0ef0867773d4de990f6a025a6946308 2023-04-24 09:00:00,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326860869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:00,873 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/2b7556f4e0ea46cd8d29e23011ed7792 2023-04-24 09:00:00,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:00,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326860875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,282 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/e8f92b0d2b4848bf82c9de3f8bdbfeb9 2023-04-24 09:00:01,285 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/a0ef0867773d4de990f6a025a6946308 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/a0ef0867773d4de990f6a025a6946308 2023-04-24 09:00:01,288 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/a0ef0867773d4de990f6a025a6946308, entries=150, sequenceid=149, filesize=25.3 K 2023-04-24 09:00:01,289 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/2b7556f4e0ea46cd8d29e23011ed7792 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2b7556f4e0ea46cd8d29e23011ed7792 2023-04-24 09:00:01,292 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2b7556f4e0ea46cd8d29e23011ed7792, entries=150, sequenceid=149, filesize=11.8 K 2023-04-24 09:00:01,293 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/e8f92b0d2b4848bf82c9de3f8bdbfeb9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/e8f92b0d2b4848bf82c9de3f8bdbfeb9 2023-04-24 09:00:01,297 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/e8f92b0d2b4848bf82c9de3f8bdbfeb9, entries=150, sequenceid=149, filesize=11.8 K 2023-04-24 09:00:01,297 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d7e0c211118009ab258bf68c7108cae8 in 1660ms, sequenceid=149, compaction requested=false 2023-04-24 09:00:01,297 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:01,297 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:01,298 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:01,298 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:01,298 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,298 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:01,298 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:01,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:01,352 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:01,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:01,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:01,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:01,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6820e973[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:01,353 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:01,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:01,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:01,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:01,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:01,360 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:01,360 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,360 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:01,360 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:01,360 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:01,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:01,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:01,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,369 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:01,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:01,370 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:01,370 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:01,370 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:01,370 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:01,370 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:01,377 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,377 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:01,377 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:01,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,378 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:01,378 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:01,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:01,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:01,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:01,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:01,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:01,385 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:01,385 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:01,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:01,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,385 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:01,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:01,386 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:01,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:01,386 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,486 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:01,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:01,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:01,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:01,488 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:01,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:01,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:01,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:01,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:01,493 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:01,493 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:01,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,494 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:01,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:01,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:01,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:01,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:01,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:01,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:01,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:01,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:01,495 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:01,495 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:01,495 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:01,496 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 09:00:01,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:01,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:01,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:01,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:01,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:01,496 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:01,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:01,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:01,908 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424f6ceffddc967467493d48a0998f353d2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f6ceffddc967467493d48a0998f353d2 2023-04-24 09:00:01,915 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=163, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/4648447d8f2c4475b295e27d4ad76147 2023-04-24 09:00:01,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:01,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326861928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:01,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326801902, max timestamp=1682326801928Num uniques 859; 2023-04-24 09:00:01,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326801902, max timestamp=1682326801928Num uniques 859; 2023-04-24 09:00:01,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326801902, max timestamp=1682326801928Num uniques 859; 2023-04-24 09:00:01,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:01,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:01,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:01,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:01,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326861990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326862031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326862045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326862046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,055 DEBUG [Thread-1989] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:02,055 DEBUG [Thread-1993] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:02,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326862049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,059 DEBUG [Thread-1995] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:02,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326862097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326862234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326862302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,325 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/f566829c327e403d81fe4c9df734fac0 2023-04-24 09:00:02,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52690 deadline: 1682326862538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:02,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52680 deadline: 1682326862609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,745 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/4f7d906d1a3249aaa83bde50fe29c27e 2023-04-24 09:00:02,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/4648447d8f2c4475b295e27d4ad76147 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/4648447d8f2c4475b295e27d4ad76147 2023-04-24 09:00:02,751 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/4648447d8f2c4475b295e27d4ad76147, entries=150, sequenceid=163, filesize=25.3 K 2023-04-24 09:00:02,752 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/f566829c327e403d81fe4c9df734fac0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f566829c327e403d81fe4c9df734fac0 2023-04-24 09:00:02,755 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f566829c327e403d81fe4c9df734fac0, entries=150, sequenceid=163, filesize=11.8 K 2023-04-24 09:00:02,756 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/4f7d906d1a3249aaa83bde50fe29c27e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/4f7d906d1a3249aaa83bde50fe29c27e 2023-04-24 09:00:02,764 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/4f7d906d1a3249aaa83bde50fe29c27e, entries=150, sequenceid=163, filesize=11.8 K 2023-04-24 09:00:02,765 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for d7e0c211118009ab258bf68c7108cae8 in 1270ms, sequenceid=163, compaction requested=true 2023-04-24 09:00:02,765 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:02,765 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:02,765 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:02,765 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:02,765 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,765 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:02,765 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:02,793 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:02,793 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:02,794 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:02,794 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:02,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:02,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c12d3c[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:02,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:02,795 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:02,802 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:02,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:02,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:02,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:02,802 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:02,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,802 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:02,810 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:02,810 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:02,810 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,810 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:02,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:02,812 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:02,812 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:02,812 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:02,812 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:02,812 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:02,818 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,818 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:02,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,818 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:02,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:02,819 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,820 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,855 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,855 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:02,855 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:02,855 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:02,855 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:02,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:02,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:02,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:02,856 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,856 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:02,856 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:02,856 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,855 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:02,856 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:02,856 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:02,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:02,956 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:02,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:02,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:02,960 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:02,960 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:02,960 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:02,960 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:02,960 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,960 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:02,968 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:02,968 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:02,969 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:02,969 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:02,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:02,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:02,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:02,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:02,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:02,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:02,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:02,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:02,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:02,970 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:02,970 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:02,970 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-24 09:00:02,971 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:02,971 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:02,971 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:02,971 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:02,971 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:02,971 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:02,989 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042462dafb7c14044dd3bd5aa6a01381313a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042462dafb7c14044dd3bd5aa6a01381313a 2023-04-24 09:00:02,995 WARN [DataStreamer for file /user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/e0dabe0251794b2eaba5d646093a4055] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-24 09:00:02,996 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=197, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/e0dabe0251794b2eaba5d646093a4055 2023-04-24 09:00:03,008 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/9bcce601e144425e83980681bcf2a32f 2023-04-24 09:00:03,020 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/fdf9f56fe91d4218b65f520de6ab5342 2023-04-24 09:00:03,023 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/e0dabe0251794b2eaba5d646093a4055 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e0dabe0251794b2eaba5d646093a4055 2023-04-24 09:00:03,029 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e0dabe0251794b2eaba5d646093a4055, entries=150, sequenceid=197, filesize=25.3 K 2023-04-24 09:00:03,030 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/9bcce601e144425e83980681bcf2a32f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/9bcce601e144425e83980681bcf2a32f 2023-04-24 09:00:03,033 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/9bcce601e144425e83980681bcf2a32f, entries=150, sequenceid=197, filesize=11.8 K 2023-04-24 09:00:03,034 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/fdf9f56fe91d4218b65f520de6ab5342 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fdf9f56fe91d4218b65f520de6ab5342 2023-04-24 09:00:03,042 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fdf9f56fe91d4218b65f520de6ab5342, entries=150, sequenceid=197, filesize=11.8 K 2023-04-24 09:00:03,043 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=0 B/0 for d7e0c211118009ab258bf68c7108cae8 in 73ms, sequenceid=197, compaction requested=true 2023-04-24 09:00:03,043 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:03,043 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:03,043 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:03,043 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:03,043 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,043 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:03,043 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:03,052 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,052 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:03,052 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:03,052 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:03,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:03,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e4b9d0[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:03,054 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:03,054 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:03,060 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:03,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:03,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:03,060 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:03,060 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:03,060 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:03,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:03,068 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:03,068 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:03,069 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:03,069 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:03,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:03,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:03,071 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:03,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:03,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:03,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:03,071 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:03,077 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,077 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:03,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,077 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:03,077 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:03,078 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,085 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:03,085 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:03,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:03,085 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:03,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:03,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:03,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:03,086 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:03,086 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:03,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:03,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:03,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:03,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:03,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:03,186 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:03,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:03,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:03,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:03,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:03,204 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:03,204 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:03,204 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:03,204 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:03,205 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,205 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:03,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:03,210 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:03,211 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,211 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:03,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:03,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:03,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:03,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:03,212 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:03,212 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:03,212 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:03,224 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:03,224 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:03,224 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:03,224 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:03,224 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 09:00:03,224 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:03,224 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:03,224 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:03,225 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:03,225 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:03,225 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:03,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:03,237 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:03,266 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304242f9f4b7355604e09b3ac6ab6da277426 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242f9f4b7355604e09b3ac6ab6da277426 2023-04-24 09:00:03,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326803320, max timestamp=1682326803358Num uniques 859; 2023-04-24 09:00:03,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326803320, max timestamp=1682326803358Num uniques 859; 2023-04-24 09:00:03,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:03,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:03,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326803320, max timestamp=1682326803358Num uniques 859; 2023-04-24 09:00:03,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:03,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:03,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326803358, max timestamp=1682326803444Num uniques 1068; 2023-04-24 09:00:03,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326803358, max timestamp=1682326803444Num uniques 1068; 2023-04-24 09:00:03,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 09:00:03,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 09:00:03,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326863437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326803358, max timestamp=1682326803444Num uniques 1068; 2023-04-24 09:00:03,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 09:00:03,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:03,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326863562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326803547, max timestamp=1682326803564Num uniques 1077; 2023-04-24 09:00:03,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 09:00:03,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326803547, max timestamp=1682326803564Num uniques 1077; 2023-04-24 09:00:03,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 09:00:03,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326803547, max timestamp=1682326803564Num uniques 1077; 2023-04-24 09:00:03,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 09:00:03,682 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=205, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/142752f799ab4675a1366292c89376f5 2023-04-24 09:00:03,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326803664, max timestamp=1682326803691Num uniques 1086; 2023-04-24 09:00:03,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:03,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326803664, max timestamp=1682326803691Num uniques 1086; 2023-04-24 09:00:03,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:03,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326803664, max timestamp=1682326803691Num uniques 1086; 2023-04-24 09:00:03,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:03,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682326803732, max timestamp=1682326803764Num uniques 1095; 2023-04-24 09:00:03,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-24 09:00:03,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682326803732, max timestamp=1682326803764Num uniques 1095; 2023-04-24 09:00:03,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1682326803732, max timestamp=1682326803764Num uniques 1095; 2023-04-24 09:00:03,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-24 09:00:03,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-24 09:00:03,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:03,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326863845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:03,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326863845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:03,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326863952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:03,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:03,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326863952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,097 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/2ab693a4b98d4deab30f86cae5f70703 2023-04-24 09:00:04,118 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/d4b30315d7704d4984e4cb0280d665d4 2023-04-24 09:00:04,123 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/142752f799ab4675a1366292c89376f5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/142752f799ab4675a1366292c89376f5 2023-04-24 09:00:04,131 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/142752f799ab4675a1366292c89376f5, entries=100, sequenceid=205, filesize=18.5 K 2023-04-24 09:00:04,136 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/2ab693a4b98d4deab30f86cae5f70703 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2ab693a4b98d4deab30f86cae5f70703 2023-04-24 09:00:04,141 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2ab693a4b98d4deab30f86cae5f70703, entries=100, sequenceid=205, filesize=9.4 K 2023-04-24 09:00:04,142 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/d4b30315d7704d4984e4cb0280d665d4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d4b30315d7704d4984e4cb0280d665d4 2023-04-24 09:00:04,146 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d4b30315d7704d4984e4cb0280d665d4, entries=100, sequenceid=205, filesize=9.4 K 2023-04-24 09:00:04,147 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for d7e0c211118009ab258bf68c7108cae8 in 923ms, sequenceid=205, compaction requested=true 2023-04-24 09:00:04,147 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:04,147 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:04,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:04,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:04,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:04,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:04,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-04-24 09:00:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:04,169 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:04,177 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,177 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,177 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,177 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:04,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2be4d67f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:04,179 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:04,179 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,181 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304246d4ce7ff24b84780a1066baf93e773d8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246d4ce7ff24b84780a1066baf93e773d8 2023-04-24 09:00:04,185 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:04,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:04,185 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:04,188 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/17fd2805c29f413db2b42577000bfb14 2023-04-24 09:00:04,188 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,188 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:04,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:04,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326864181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,193 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,194 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:04,194 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:04,194 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:04,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:04,197 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:04,197 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:04,197 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:04,197 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:04,197 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:04,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:04,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326864190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,202 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,202 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,202 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:04,202 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,203 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:04,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:04,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:04,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:04,210 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,211 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,210 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:04,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:04,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:04,214 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:04,214 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:04,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:04,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326864291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:04,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326864300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,314 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:04,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:04,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:04,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:04,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:04,317 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:04,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:04,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:04,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:04,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,317 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,327 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:04,327 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:04,327 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,327 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:04,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:04,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:04,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:04,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:04,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:04,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:04,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:04,328 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:04,328 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,328 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:04,329 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:04,329 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:04,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:04,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:04,329 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:04,335 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,335 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,335 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,335 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:04,337 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:04,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4460c901[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:04,337 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,343 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,344 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,344 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:04,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:04,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:04,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:04,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:04,352 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,352 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:04,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:04,354 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:04,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:04,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:04,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:04,354 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:04,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,360 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:04,360 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,361 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,368 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,368 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:04,368 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:04,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:04,368 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:04,368 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:04,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:04,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:04,368 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,369 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:04,369 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:04,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:04,469 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:04,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:04,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:04,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:04,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:04,476 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:04,476 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:04,476 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:04,476 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:04,476 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,476 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:04,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326864498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:04,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326864507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,522 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:04,523 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:04,523 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,523 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:04,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:04,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:04,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:04,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:04,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:04,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:04,524 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:04,524 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,524 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:04,524 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:04,525 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:04,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:04,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:04,525 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:04,539 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:04,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c40b47c[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:04,540 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:04,541 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,547 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,548 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:04,548 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:04,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:04,548 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:04,556 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:04,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:04,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:04,559 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:04,560 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:04,560 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:04,560 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:04,560 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:04,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,573 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:04,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:04,573 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,574 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,589 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,589 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:04,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:04,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:04,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:04,589 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:04,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:04,590 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:04,590 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:04,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:04,589 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,590 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:04,590 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,590 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,590 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:04,590 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/1fc24fde52f64411ab3713a5e71eba2f 2023-04-24 09:00:04,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/a9d00784f5304fd7b01d1bb460f6ea13 2023-04-24 09:00:04,627 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/17fd2805c29f413db2b42577000bfb14 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/17fd2805c29f413db2b42577000bfb14 2023-04-24 09:00:04,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/17fd2805c29f413db2b42577000bfb14, entries=200, sequenceid=311, filesize=32.3 K 2023-04-24 09:00:04,630 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/1fc24fde52f64411ab3713a5e71eba2f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1fc24fde52f64411ab3713a5e71eba2f 2023-04-24 09:00:04,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1fc24fde52f64411ab3713a5e71eba2f, entries=150, sequenceid=311, filesize=11.9 K 2023-04-24 09:00:04,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/a9d00784f5304fd7b01d1bb460f6ea13 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a9d00784f5304fd7b01d1bb460f6ea13 2023-04-24 09:00:04,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a9d00784f5304fd7b01d1bb460f6ea13, entries=150, sequenceid=311, filesize=11.9 K 2023-04-24 09:00:04,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.95 KB/442320, currentSize=26.84 KB/27480 for d7e0c211118009ab258bf68c7108cae8 in 469ms, sequenceid=311, compaction requested=true 2023-04-24 09:00:04,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:04,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:04,639 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 09:00:04,639 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 09:00:04,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:04,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:04,644 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70096 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 09:00:04,644 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/B is initiating minor compaction (all files) 2023-04-24 09:00:04,644 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/B in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,644 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/312ccc8776f04a55aeb90a2d840e317b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2b7556f4e0ea46cd8d29e23011ed7792, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f566829c327e403d81fe4c9df734fac0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/9bcce601e144425e83980681bcf2a32f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2ab693a4b98d4deab30f86cae5f70703, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1fc24fde52f64411ab3713a5e71eba2f] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=68.5 K 2023-04-24 09:00:04,644 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155692 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 09:00:04,645 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/A is initiating minor compaction (all files) 2023-04-24 09:00:04,645 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/A in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,645 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cf211ad810d14cb9bd088049ca6b4415, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/a0ef0867773d4de990f6a025a6946308, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/4648447d8f2c4475b295e27d4ad76147, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e0dabe0251794b2eaba5d646093a4055, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/142752f799ab4675a1366292c89376f5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/17fd2805c29f413db2b42577000bfb14] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=152.0 K 2023-04-24 09:00:04,645 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 312ccc8776f04a55aeb90a2d840e317b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326795598 2023-04-24 09:00:04,645 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 2b7556f4e0ea46cd8d29e23011ed7792, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682326797888 2023-04-24 09:00:04,645 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting cf211ad810d14cb9bd088049ca6b4415, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326795598 2023-04-24 09:00:04,645 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f566829c327e403d81fe4c9df734fac0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682326799750 2023-04-24 09:00:04,646 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 9bcce601e144425e83980681bcf2a32f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682326801968 2023-04-24 09:00:04,646 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting a0ef0867773d4de990f6a025a6946308, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682326797888 2023-04-24 09:00:04,646 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 2ab693a4b98d4deab30f86cae5f70703, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1682326803199 2023-04-24 09:00:04,646 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 4648447d8f2c4475b295e27d4ad76147, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682326799750 2023-04-24 09:00:04,646 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 1fc24fde52f64411ab3713a5e71eba2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326803830 2023-04-24 09:00:04,646 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e0dabe0251794b2eaba5d646093a4055, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682326801968 2023-04-24 09:00:04,647 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 142752f799ab4675a1366292c89376f5, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1682326803199 2023-04-24 09:00:04,647 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 17fd2805c29f413db2b42577000bfb14, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326803829 2023-04-24 09:00:04,668 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#B#compaction#300 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:04,675 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#A#compaction#301 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:04,680 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/e84e9cd62db54fb6bce3da99a09f71af as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e84e9cd62db54fb6bce3da99a09f71af 2023-04-24 09:00:04,690 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:04,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:04,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:04,716 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/B of d7e0c211118009ab258bf68c7108cae8 into e84e9cd62db54fb6bce3da99a09f71af(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:04,716 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:04,716 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/B, priority=10, startTime=1682326804638; duration=0sec 2023-04-24 09:00:04,716 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:04,716 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-24 09:00:04,718 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70096 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-24 09:00:04,718 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/C is initiating minor compaction (all files) 2023-04-24 09:00:04,719 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/C in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,719 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/540671d3bfe347cdb676c4872007fdd3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/e8f92b0d2b4848bf82c9de3f8bdbfeb9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/4f7d906d1a3249aaa83bde50fe29c27e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fdf9f56fe91d4218b65f520de6ab5342, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d4b30315d7704d4984e4cb0280d665d4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a9d00784f5304fd7b01d1bb460f6ea13] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=68.5 K 2023-04-24 09:00:04,719 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 540671d3bfe347cdb676c4872007fdd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1682326795598 2023-04-24 09:00:04,719 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e8f92b0d2b4848bf82c9de3f8bdbfeb9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682326797888 2023-04-24 09:00:04,720 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4f7d906d1a3249aaa83bde50fe29c27e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682326799750 2023-04-24 09:00:04,720 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting fdf9f56fe91d4218b65f520de6ab5342, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1682326801968 2023-04-24 09:00:04,720 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d4b30315d7704d4984e4cb0280d665d4, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1682326803199 2023-04-24 09:00:04,721 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a9d00784f5304fd7b01d1bb460f6ea13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326803830 2023-04-24 09:00:04,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:04,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:04,724 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:04,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:04,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:04,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:04,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,731 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:04,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:04,731 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:04,731 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:04,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:04,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:04,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:04,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:04,733 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:04,733 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:04,733 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:04,734 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:04,734 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:04,734 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:04,734 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:04,734 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 09:00:04,738 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:04,738 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:04,739 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/cbe877dbc4ed4e54a9cb3960ec7f0f9e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cbe877dbc4ed4e54a9cb3960ec7f0f9e 2023-04-24 09:00:04,739 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:04,739 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:04,739 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:04,739 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:04,742 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#C#compaction#302 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:04,748 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/A of d7e0c211118009ab258bf68c7108cae8 into cbe877dbc4ed4e54a9cb3960ec7f0f9e(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:04,748 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:04,748 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/A, priority=10, startTime=1682326804638; duration=0sec 2023-04-24 09:00:04,749 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:04,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:04,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:04,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326804883, max timestamp=1682326804906Num uniques 859; 2023-04-24 09:00:04,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:04,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682326804883, max timestamp=1682326804906Num uniques 859; 2023-04-24 09:00:04,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-24 09:00:04,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682326804883, max timestamp=1682326804906Num uniques 859; 2023-04-24 09:00:04,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-24 09:00:04,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682326804957, max timestamp=1682326804974Num uniques 1009; 2023-04-24 09:00:04,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-24 09:00:04,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682326804957, max timestamp=1682326804974Num uniques 1018; 2023-04-24 09:00:04,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-24 09:00:04,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682326804957, max timestamp=1682326804974Num uniques 1009; 2023-04-24 09:00:04,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-24 09:00:05,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326805048, max timestamp=1682326805088Num uniques 1027; 2023-04-24 09:00:05,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326805048, max timestamp=1682326805088Num uniques 1027; 2023-04-24 09:00:05,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 09:00:05,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 09:00:05,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326805048, max timestamp=1682326805088Num uniques 1027; 2023-04-24 09:00:05,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-24 09:00:05,156 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/3238effc6fb348429f17c9009f122103 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/3238effc6fb348429f17c9009f122103 2023-04-24 09:00:05,164 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304246df696cd3de84133a091b015e5bc2c17 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246df696cd3de84133a091b015e5bc2c17 2023-04-24 09:00:05,169 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/C of d7e0c211118009ab258bf68c7108cae8 into 3238effc6fb348429f17c9009f122103(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:05,169 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:05,169 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/C, priority=10, startTime=1682326804639; duration=0sec 2023-04-24 09:00:05,169 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:05,173 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=319, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/78a99ac0e9d5432eb39cc7d1751131a4 2023-04-24 09:00:05,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326805113, max timestamp=1682326805203Num uniques 1086; 2023-04-24 09:00:05,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:05,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326805113, max timestamp=1682326805203Num uniques 1086; 2023-04-24 09:00:05,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:05,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326805113, max timestamp=1682326805203Num uniques 1086; 2023-04-24 09:00:05,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:05,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326805254, max timestamp=1682326805282Num uniques 1095; 2023-04-24 09:00:05,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 09:00:05,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326805254, max timestamp=1682326805282Num uniques 1095; 2023-04-24 09:00:05,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 09:00:05,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682326805254, max timestamp=1682326805282Num uniques 1095; 2023-04-24 09:00:05,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-24 09:00:05,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:05,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326865354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:05,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682326805322, max timestamp=1682326805358Num uniques 1054; 2023-04-24 09:00:05,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-24 09:00:05,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682326805322, max timestamp=1682326805358Num uniques 1104; 2023-04-24 09:00:05,362 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-24 09:00:05,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682326805322, max timestamp=1682326805358Num uniques 1104; 2023-04-24 09:00:05,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-24 09:00:05,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682326805438, max timestamp=1682326805470Num uniques 1063; 2023-04-24 09:00:05,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682326805438, max timestamp=1682326805470Num uniques 1063; 2023-04-24 09:00:05,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-24 09:00:05,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682326805438, max timestamp=1682326805470Num uniques 1063; 2023-04-24 09:00:05,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-24 09:00:05,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-24 09:00:05,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326865523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:05,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326865529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:05,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326805517, max timestamp=1682326805527Num uniques 1122; 2023-04-24 09:00:05,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326805517, max timestamp=1682326805527Num uniques 1122; 2023-04-24 09:00:05,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326805517, max timestamp=1682326805527Num uniques 1122; 2023-04-24 09:00:05,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-24 09:00:05,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-24 09:00:05,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-24 09:00:05,597 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/a6bc518a510040dabf6d9b69860712cc 2023-04-24 09:00:05,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682326805649, max timestamp=1682326805675Num uniques 1031; 2023-04-24 09:00:05,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 09:00:05,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682326805649, max timestamp=1682326805675Num uniques 1031; 2023-04-24 09:00:05,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 09:00:05,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682326805649, max timestamp=1682326805675Num uniques 1031; 2023-04-24 09:00:05,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 09:00:05,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682326805702, max timestamp=1682326805721Num uniques 1090; 2023-04-24 09:00:05,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682326805702, max timestamp=1682326805721Num uniques 1090; 2023-04-24 09:00:05,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-24 09:00:05,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682326805702, max timestamp=1682326805721Num uniques 1090; 2023-04-24 09:00:05,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-24 09:00:05,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-24 09:00:05,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682326805759, max timestamp=1682326805778Num uniques 1049; 2023-04-24 09:00:05,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-24 09:00:05,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682326805759, max timestamp=1682326805778Num uniques 1049; 2023-04-24 09:00:05,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-24 09:00:05,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682326805759, max timestamp=1682326805778Num uniques 1049; 2023-04-24 09:00:05,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-24 09:00:05,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682326805821, max timestamp=1682326805834Num uniques 1058; 2023-04-24 09:00:05,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682326805821, max timestamp=1682326805834Num uniques 1058; 2023-04-24 09:00:05,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-24 09:00:05,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-24 09:00:05,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682326805821, max timestamp=1682326805834Num uniques 1058; 2023-04-24 09:00:05,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-24 09:00:05,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682326805866, max timestamp=1682326805878Num uniques 1017; 2023-04-24 09:00:05,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-24 09:00:05,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682326805866, max timestamp=1682326805878Num uniques 1017; 2023-04-24 09:00:05,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-24 09:00:05,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682326805866, max timestamp=1682326805878Num uniques 1017; 2023-04-24 09:00:05,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-24 09:00:05,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682326805911, max timestamp=1682326805925Num uniques 1076; 2023-04-24 09:00:05,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-24 09:00:05,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682326805911, max timestamp=1682326805925Num uniques 1076; 2023-04-24 09:00:05,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-24 09:00:05,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682326805911, max timestamp=1682326805925Num uniques 1076; 2023-04-24 09:00:05,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-24 09:00:05,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682326805958, max timestamp=1682326805971Num uniques 1035; 2023-04-24 09:00:05,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-24 09:00:05,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682326805958, max timestamp=1682326805971Num uniques 1035; 2023-04-24 09:00:05,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-24 09:00:05,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682326805958, max timestamp=1682326805971Num uniques 1035; 2023-04-24 09:00:05,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-24 09:00:06,006 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/38fa2091a79740bb83a175adbdebb97d 2023-04-24 09:00:06,009 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/78a99ac0e9d5432eb39cc7d1751131a4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/78a99ac0e9d5432eb39cc7d1751131a4 2023-04-24 09:00:06,011 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/78a99ac0e9d5432eb39cc7d1751131a4, entries=100, sequenceid=319, filesize=18.6 K 2023-04-24 09:00:06,012 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/a6bc518a510040dabf6d9b69860712cc as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a6bc518a510040dabf6d9b69860712cc 2023-04-24 09:00:06,016 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a6bc518a510040dabf6d9b69860712cc, entries=100, sequenceid=319, filesize=9.5 K 2023-04-24 09:00:06,016 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/38fa2091a79740bb83a175adbdebb97d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/38fa2091a79740bb83a175adbdebb97d 2023-04-24 09:00:06,019 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/38fa2091a79740bb83a175adbdebb97d, entries=100, sequenceid=319, filesize=9.5 K 2023-04-24 09:00:06,020 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=127.47 KB/130530 for d7e0c211118009ab258bf68c7108cae8 in 1286ms, sequenceid=319, compaction requested=false 2023-04-24 09:00:06,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:06,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:06,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:06,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:06,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:06,022 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=134.18 KB heapSize=377.86 KB 2023-04-24 09:00:06,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:06,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:06,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:06,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:06,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:06,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:06,031 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,031 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,031 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,031 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:06,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:06,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8aae1d[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:06,032 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:06,032 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,039 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:06,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:06,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:06,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:06,048 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:06,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:06,050 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:06,050 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:06,050 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:06,050 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:06,050 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:06,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,056 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326866047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326866048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,057 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,064 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,064 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,064 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:06,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,064 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,064 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,064 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:06,064 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,064 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:06,064 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,065 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52692 deadline: 1682326866057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,068 DEBUG [Thread-1989] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:06,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52668 deadline: 1682326866059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,072 DEBUG [Thread-1993] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:06,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:52634 deadline: 1682326866079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,090 DEBUG [Thread-1995] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:06,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326866158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326866158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,165 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:06,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:06,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:06,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:06,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:06,167 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:06,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:06,167 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:06,168 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,168 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,172 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,173 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,173 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:06,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:06,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:06,174 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,174 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:06,174 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:06,174 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:06,181 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,182 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,182 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:06,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:06,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d0c2dd[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:06,189 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:06,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:06,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:06,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:06,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,197 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,197 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:06,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:06,200 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:06,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:06,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:06,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:06,200 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:06,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,207 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,208 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:06,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:06,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:06,222 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:06,222 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:06,322 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:06,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:06,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:06,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:06,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:06,324 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:06,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:06,325 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:06,326 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,326 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,331 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,331 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,331 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,331 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:06,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:06,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:06,333 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:06,333 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,333 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:06,333 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:06,334 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:06,334 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:06,358 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,360 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,361 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:06,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:06,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:06,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633a3f27[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:06,362 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326866363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326866364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,373 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:06,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:06,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:06,373 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,373 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:06,381 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:06,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:06,384 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:06,384 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:06,384 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:06,384 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:06,384 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:06,395 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,396 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,396 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,396 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,397 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,407 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:06,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:06,407 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:06,407 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:06,448 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304240032fddb90b84865a8f07d1427adc8d6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240032fddb90b84865a8f07d1427adc8d6 2023-04-24 09:00:06,468 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=597, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/0c0f65a576934c82a6c7c54f216c950d 2023-04-24 09:00:06,507 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:06,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:06,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:06,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:06,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:06,510 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:06,510 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:06,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:06,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,511 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,522 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:06,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:06,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:06,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:06,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:06,529 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:06,529 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,529 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:06,530 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:06,530 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:06,530 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,530 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:06,530 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:06,530 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,530 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:06,530 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:06,530 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,530 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:06,530 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:06,539 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:06,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a908468[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:06,542 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:06,542 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,548 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,548 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:06,548 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:06,548 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:06,548 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,549 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:06,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:06,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:06,556 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:06,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:06,558 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:06,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:06,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:06,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:06,558 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:06,564 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,564 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,564 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,565 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,565 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,573 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,573 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,573 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,573 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:06,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:06,573 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,573 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,573 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:06,573 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:06,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:06,673 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:06,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:06,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:06,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:06,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:06,676 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:06,676 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,676 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:06,676 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:06,676 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,677 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326866668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:06,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326866670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,681 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,681 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,681 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,681 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:06,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:06,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:06,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:06,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:06,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,682 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:06,688 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,688 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:06,688 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:06,688 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:06,688 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:06,697 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:06,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53933d61[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:06,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:06,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,706 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,706 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,706 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:06,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:06,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:06,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:06,714 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:06,714 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,714 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,714 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:06,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:06,716 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:06,716 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:06,716 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:06,716 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:06,716 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:06,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,723 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,731 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:06,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,731 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,731 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:06,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,832 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:06,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:06,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:06,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:06,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:06,834 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:06,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:06,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:06,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,834 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,839 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,839 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,839 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,839 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:06,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:06,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:06,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:06,840 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:06,840 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:06,840 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:06,840 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:06,840 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:06,840 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,841 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:06,841 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:06,841 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:06,841 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:06,841 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:06,841 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,841 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:06,841 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:06,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,847 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,848 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,848 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:06,849 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:06,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7472cbe4[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:06,849 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,856 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:06,856 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,856 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:06,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:06,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:06,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/ea7aa5fa7d6b4cef96c13dad150b472c 2023-04-24 09:00:06,884 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,884 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:06,884 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,884 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:06,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:06,887 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:06,887 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:06,887 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:06,887 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:06,887 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:06,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:06,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:06,899 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,900 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:06,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:06,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:06,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:06,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:06,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:06,906 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:06,906 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:06,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:06,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,006 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:07,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:07,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:07,009 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:07,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:07,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:07,010 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,010 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,014 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:07,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:07,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:07,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:07,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:07,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:07,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:07,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:07,016 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,016 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:07,016 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:07,016 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:07,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:07,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:07,016 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:07,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,022 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:07,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4603114a[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:07,024 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:07,024 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,031 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,031 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:07,031 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:07,031 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,031 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:07,031 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:07,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:07,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,039 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:07,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:07,041 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:07,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:07,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:07,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:07,041 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:07,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,048 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,050 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,056 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:07,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,056 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:07,056 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:07,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,156 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:07,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:07,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:07,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:07,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:07,159 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:07,159 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,159 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:07,159 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:07,159 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,160 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,165 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,165 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:07,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:07,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:07,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:07,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:07,166 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,166 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:07,166 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:07,166 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:07,166 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:07,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,172 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,173 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,173 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:07,174 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:07,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@681e6e26[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:07,174 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,181 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:07,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:07,181 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,181 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:07,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:07,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:07,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326867179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:07,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326867181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:07,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,189 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:07,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:07,191 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:07,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:07,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:07,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:07,191 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:07,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,198 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,206 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:07,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:07,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:07,206 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:07,206 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:07,207 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,207 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=597 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/232bfa736eb144baa18c1d047f8e4629 2023-04-24 09:00:07,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/0c0f65a576934c82a6c7c54f216c950d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0c0f65a576934c82a6c7c54f216c950d 2023-04-24 09:00:07,306 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:07,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:07,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0c0f65a576934c82a6c7c54f216c950d, entries=200, sequenceid=597, filesize=32.3 K 2023-04-24 09:00:07,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/ea7aa5fa7d6b4cef96c13dad150b472c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/ea7aa5fa7d6b4cef96c13dad150b472c 2023-04-24 09:00:07,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:07,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:07,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:07,310 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:07,310 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,310 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:07,310 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:07,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/ea7aa5fa7d6b4cef96c13dad150b472c, entries=150, sequenceid=597, filesize=11.9 K 2023-04-24 09:00:07,312 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,312 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/232bfa736eb144baa18c1d047f8e4629 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/232bfa736eb144baa18c1d047f8e4629 2023-04-24 09:00:07,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/232bfa736eb144baa18c1d047f8e4629, entries=150, sequenceid=597, filesize=11.9 K 2023-04-24 09:00:07,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~377.81 KB/386880, currentSize=46.96 KB/48090 for d7e0c211118009ab258bf68c7108cae8 in 1294ms, sequenceid=597, compaction requested=true 2023-04-24 09:00:07,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:07,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:07,316 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:07,316 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:07,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:07,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:07,317 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78608 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:07,317 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/A is initiating minor compaction (all files) 2023-04-24 09:00:07,317 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/A in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,317 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cbe877dbc4ed4e54a9cb3960ec7f0f9e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/78a99ac0e9d5432eb39cc7d1751131a4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0c0f65a576934c82a6c7c54f216c950d] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=76.8 K 2023-04-24 09:00:07,318 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:07,318 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting cbe877dbc4ed4e54a9cb3960ec7f0f9e, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326803830 2023-04-24 09:00:07,318 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/B is initiating minor compaction (all files) 2023-04-24 09:00:07,318 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/B in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,318 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e84e9cd62db54fb6bce3da99a09f71af, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a6bc518a510040dabf6d9b69860712cc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/ea7aa5fa7d6b4cef96c13dad150b472c] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=33.8 K 2023-04-24 09:00:07,318 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 78a99ac0e9d5432eb39cc7d1751131a4, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682326804179 2023-04-24 09:00:07,318 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e84e9cd62db54fb6bce3da99a09f71af, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326803830 2023-04-24 09:00:07,319 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a6bc518a510040dabf6d9b69860712cc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682326804179 2023-04-24 09:00:07,319 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 0c0f65a576934c82a6c7c54f216c950d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1682326806004 2023-04-24 09:00:07,319 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting ea7aa5fa7d6b4cef96c13dad150b472c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1682326806011 2023-04-24 09:00:07,320 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,321 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,321 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,321 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:07,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:07,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:07,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:07,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:07,331 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:07,331 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,331 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:07,336 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#B#compaction#310 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:07,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:07,336 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:07,336 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,337 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:07,337 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 09:00:07,337 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:07,337 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:07,337 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:07,337 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:07,337 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:07,337 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:07,349 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#A#compaction#309 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:07,379 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424171e2f35d6964eb39c78f5157ef57367 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424171e2f35d6964eb39c78f5157ef57367 2023-04-24 09:00:07,406 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=607, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/2a5412353e574139b0cc35d55533f7ba 2023-04-24 09:00:07,418 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/f51495e98a2442b3a67dcfd62152121f 2023-04-24 09:00:07,445 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/825d4c2c81bc492a8d1e24e417c37ca8 2023-04-24 09:00:07,449 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/2a5412353e574139b0cc35d55533f7ba as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/2a5412353e574139b0cc35d55533f7ba 2023-04-24 09:00:07,453 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/2a5412353e574139b0cc35d55533f7ba, entries=100, sequenceid=607, filesize=18.6 K 2023-04-24 09:00:07,454 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/f51495e98a2442b3a67dcfd62152121f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f51495e98a2442b3a67dcfd62152121f 2023-04-24 09:00:07,474 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f51495e98a2442b3a67dcfd62152121f, entries=100, sequenceid=607, filesize=9.5 K 2023-04-24 09:00:07,477 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/825d4c2c81bc492a8d1e24e417c37ca8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/825d4c2c81bc492a8d1e24e417c37ca8 2023-04-24 09:00:07,480 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/825d4c2c81bc492a8d1e24e417c37ca8, entries=100, sequenceid=607, filesize=9.5 K 2023-04-24 09:00:07,481 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for d7e0c211118009ab258bf68c7108cae8 in 144ms, sequenceid=607, compaction requested=true 2023-04-24 09:00:07,481 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:07,481 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,481 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:07,481 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:07,481 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,481 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:07,481 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:07,489 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,489 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,489 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:07,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@222c3e71[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:07,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:07,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:07,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:07,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:07,498 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,498 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:07,506 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:07,506 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,506 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,506 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:07,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:07,509 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:07,509 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:07,509 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:07,509 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:07,509 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:07,535 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,535 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,535 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,536 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,537 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,554 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,554 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,554 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,554 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:07,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:07,554 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,554 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,554 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,555 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,555 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,554 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,555 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:07,555 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:07,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:07,655 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:07,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:07,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:07,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:07,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:07,668 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:07,668 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,668 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:07,668 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:07,668 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,668 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,681 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,681 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,681 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,681 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:07,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:07,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:07,692 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:07,692 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,692 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:07,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:07,700 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:07,700 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:07,700 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:07,700 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:07,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:07,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:07,700 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:07,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,706 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:07,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4063ec3b[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:07,708 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:07,708 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,714 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:07,714 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:07,715 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:07,715 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:07,722 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:07,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:07,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:07,725 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:07,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:07,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:07,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:07,725 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:07,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,731 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,732 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,733 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,739 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,739 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,739 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,739 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,739 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:07,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:07,740 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:07,740 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:07,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:07,739 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,739 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,740 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,748 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/1c8af361aabe4b9fb5afec7505231310 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1c8af361aabe4b9fb5afec7505231310 2023-04-24 09:00:07,759 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/B of d7e0c211118009ab258bf68c7108cae8 into 1c8af361aabe4b9fb5afec7505231310(size=12.5 K), total size for store is 22.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:07,759 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:07,759 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/B, priority=13, startTime=1682326807316; duration=0sec 2023-04-24 09:00:07,759 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:07,759 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:07,761 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44334 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:07,761 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/C is initiating minor compaction (all files) 2023-04-24 09:00:07,761 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/C in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,761 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/3238effc6fb348429f17c9009f122103, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/38fa2091a79740bb83a175adbdebb97d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/232bfa736eb144baa18c1d047f8e4629, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/825d4c2c81bc492a8d1e24e417c37ca8] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=43.3 K 2023-04-24 09:00:07,761 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3238effc6fb348429f17c9009f122103, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682326803830 2023-04-24 09:00:07,762 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 38fa2091a79740bb83a175adbdebb97d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1682326804179 2023-04-24 09:00:07,762 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 232bfa736eb144baa18c1d047f8e4629, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1682326806011 2023-04-24 09:00:07,762 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 825d4c2c81bc492a8d1e24e417c37ca8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1682326806045 2023-04-24 09:00:07,768 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#C#compaction#314 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:07,780 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/768c0d4233ad4660af5e164b10182612 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/768c0d4233ad4660af5e164b10182612 2023-04-24 09:00:07,784 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/C of d7e0c211118009ab258bf68c7108cae8 into 768c0d4233ad4660af5e164b10182612(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:07,784 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:07,784 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/C, priority=12, startTime=1682326807316; duration=0sec 2023-04-24 09:00:07,784 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:07,840 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:07,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:07,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:07,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:07,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:07,843 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:07,843 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,843 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:07,843 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:07,844 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,844 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,854 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,854 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,854 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,854 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:07,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:07,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:07,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:07,855 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:07,855 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:07,855 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:07,856 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,856 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:07,856 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:07,856 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:07,856 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:07,864 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,864 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:07,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb43c3c[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:07,866 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:07,866 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,872 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:07,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:07,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:07,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:07,881 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,881 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:07,881 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,881 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:07,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:07,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:07,883 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:07,883 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:07,883 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:07,883 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:07,883 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:07,889 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,889 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,889 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:07,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,890 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:07,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,898 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,898 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:07,898 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:07,898 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:07,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:07,898 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:07,898 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:07,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:07,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:07,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:07,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:07,899 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:07,899 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:07,899 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:07,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:07,999 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:08,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:08,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:08,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:08,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:08,002 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:08,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:08,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:08,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:08,002 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,003 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:08,012 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:08,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:08,012 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:08,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:08,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:08,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:08,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:08,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:08,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:08,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:08,020 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:08,020 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:08,031 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,031 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:08,031 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:08,032 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:08,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:08,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c01e92[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:08,033 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:08,033 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:08,039 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:08,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:08,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:08,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:08,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:08,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:08,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:08,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:08,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:08,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:08,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:08,050 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:08,050 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:08,050 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:08,050 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:08,050 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:08,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:08,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:08,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:08,056 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:08,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:08,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:08,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:08,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:08,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,086 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:08,086 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:08,086 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:08,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:08,086 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:08,186 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:08,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:08,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:08,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:08,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:08,198 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:08,198 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:08,198 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:08,198 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:08,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:08,200 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/892c0042acc741889a29c8d20e644d29 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/892c0042acc741889a29c8d20e644d29 2023-04-24 09:00:08,203 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/A of d7e0c211118009ab258bf68c7108cae8 into 892c0042acc741889a29c8d20e644d29(size=26.0 K), total size for store is 44.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:08,203 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:08,203 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/A, priority=13, startTime=1682326807316; duration=0sec 2023-04-24 09:00:08,204 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:08,206 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,206 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:08,206 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:08,207 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:08,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:08,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:08,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:08,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:08,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:08,209 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:08,209 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:08,209 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:08,209 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:08,209 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:08,209 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:08,209 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:08,209 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-24 09:00:08,210 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:08,210 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:08,210 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:08,210 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:08,210 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:08,210 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:08,219 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042442676c35a5a74124bbb4c192ef612d75 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042442676c35a5a74124bbb4c192ef612d75 2023-04-24 09:00:08,224 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=618, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/41262fc129654adaa08b72491dc4325d 2023-04-24 09:00:08,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:08,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:08,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326808297, max timestamp=1682326808316Num uniques 859; 2023-04-24 09:00:08,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:08,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326808297, max timestamp=1682326808316Num uniques 859; 2023-04-24 09:00:08,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326808297, max timestamp=1682326808316Num uniques 859; 2023-04-24 09:00:08,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:08,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:08,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326808346, max timestamp=1682326808364Num uniques 1068; 2023-04-24 09:00:08,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 09:00:08,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326808346, max timestamp=1682326808364Num uniques 1068; 2023-04-24 09:00:08,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 09:00:08,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326808346, max timestamp=1682326808364Num uniques 1068; 2023-04-24 09:00:08,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-24 09:00:08,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326808399, max timestamp=1682326808412Num uniques 1077; 2023-04-24 09:00:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 09:00:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326808399, max timestamp=1682326808412Num uniques 1077; 2023-04-24 09:00:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 09:00:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682326808399, max timestamp=1682326808412Num uniques 1077; 2023-04-24 09:00:08,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-24 09:00:08,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326808443, max timestamp=1682326808466Num uniques 1086; 2023-04-24 09:00:08,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:08,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326808443, max timestamp=1682326808466Num uniques 1086; 2023-04-24 09:00:08,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:08,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326808443, max timestamp=1682326808466Num uniques 1086; 2023-04-24 09:00:08,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-24 09:00:08,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326808491, max timestamp=1682326808515Num uniques 1095; 2023-04-24 09:00:08,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 09:00:08,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326808491, max timestamp=1682326808515Num uniques 1095; 2023-04-24 09:00:08,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 09:00:08,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326808491, max timestamp=1682326808515Num uniques 1095; 2023-04-24 09:00:08,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-24 09:00:08,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682326808545, max timestamp=1682326808562Num uniques 1104; 2023-04-24 09:00:08,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 09:00:08,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682326808545, max timestamp=1682326808562Num uniques 1104; 2023-04-24 09:00:08,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 09:00:08,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682326808545, max timestamp=1682326808562Num uniques 1104; 2023-04-24 09:00:08,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-24 09:00:08,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682326808580, max timestamp=1682326808610Num uniques 1013; 2023-04-24 09:00:08,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 09:00:08,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682326808580, max timestamp=1682326808610Num uniques 1013; 2023-04-24 09:00:08,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682326808580, max timestamp=1682326808610Num uniques 1013; 2023-04-24 09:00:08,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 09:00:08,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-24 09:00:08,653 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/c91b2057420a465098ba9bab2d33b57b 2023-04-24 09:00:08,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326808628, max timestamp=1682326808655Num uniques 1022; 2023-04-24 09:00:08,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-24 09:00:08,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326808628, max timestamp=1682326808655Num uniques 1022; 2023-04-24 09:00:08,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-24 09:00:08,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326808628, max timestamp=1682326808655Num uniques 1022; 2023-04-24 09:00:08,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-24 09:00:08,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682326808675, max timestamp=1682326808725Num uniques 1031; 2023-04-24 09:00:08,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 09:00:08,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682326808675, max timestamp=1682326808725Num uniques 1031; 2023-04-24 09:00:08,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 09:00:08,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682326808675, max timestamp=1682326808725Num uniques 1031; 2023-04-24 09:00:08,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-24 09:00:08,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682326808750, max timestamp=1682326808770Num uniques 1090; 2023-04-24 09:00:08,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-24 09:00:08,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682326808750, max timestamp=1682326808770Num uniques 1090; 2023-04-24 09:00:08,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-24 09:00:08,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682326808750, max timestamp=1682326808770Num uniques 1090; 2023-04-24 09:00:08,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-24 09:00:08,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682326808796, max timestamp=1682326808823Num uniques 1099; 2023-04-24 09:00:08,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682326808796, max timestamp=1682326808823Num uniques 1099; 2023-04-24 09:00:08,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682326808796, max timestamp=1682326808823Num uniques 1099; 2023-04-24 09:00:08,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-24 09:00:08,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-24 09:00:08,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-24 09:00:08,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326808881, max timestamp=1682326808903Num uniques 1158; 2023-04-24 09:00:08,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-24 09:00:08,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326808881, max timestamp=1682326808903Num uniques 1158; 2023-04-24 09:00:08,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326808881, max timestamp=1682326808903Num uniques 1158; 2023-04-24 09:00:08,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-24 09:00:08,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-24 09:00:08,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682326808930, max timestamp=1682326808941Num uniques 1067; 2023-04-24 09:00:08,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-24 09:00:08,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682326808930, max timestamp=1682326808941Num uniques 1067; 2023-04-24 09:00:08,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-24 09:00:08,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682326808930, max timestamp=1682326808941Num uniques 1067; 2023-04-24 09:00:08,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-24 09:00:08,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682326808974, max timestamp=1682326808989Num uniques 1026; 2023-04-24 09:00:08,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682326808974, max timestamp=1682326808989Num uniques 1026; 2023-04-24 09:00:08,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-24 09:00:08,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-24 09:00:08,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682326808974, max timestamp=1682326808989Num uniques 1026; 2023-04-24 09:00:08,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-24 09:00:09,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682326809036, max timestamp=1682326809055Num uniques 1085; 2023-04-24 09:00:09,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682326809036, max timestamp=1682326809055Num uniques 1085; 2023-04-24 09:00:09,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-24 09:00:09,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-24 09:00:09,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1682326809036, max timestamp=1682326809055Num uniques 1085; 2023-04-24 09:00:09,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-24 09:00:09,062 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/afd9a45e58704baeb6d712c06125346c 2023-04-24 09:00:09,069 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/41262fc129654adaa08b72491dc4325d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/41262fc129654adaa08b72491dc4325d 2023-04-24 09:00:09,072 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/41262fc129654adaa08b72491dc4325d, entries=100, sequenceid=618, filesize=18.6 K 2023-04-24 09:00:09,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/c91b2057420a465098ba9bab2d33b57b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/c91b2057420a465098ba9bab2d33b57b 2023-04-24 09:00:09,078 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/c91b2057420a465098ba9bab2d33b57b, entries=100, sequenceid=618, filesize=9.5 K 2023-04-24 09:00:09,078 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/afd9a45e58704baeb6d712c06125346c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/afd9a45e58704baeb6d712c06125346c 2023-04-24 09:00:09,081 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/afd9a45e58704baeb6d712c06125346c, entries=100, sequenceid=618, filesize=9.5 K 2023-04-24 09:00:09,082 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=80.51 KB/82440 for d7e0c211118009ab258bf68c7108cae8 in 873ms, sequenceid=618, compaction requested=true 2023-04-24 09:00:09,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:09,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:09,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:09,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:09,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:09,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:09,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=87.22 KB heapSize=238.41 KB 2023-04-24 09:00:09,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:09,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:09,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:09,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:09,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:09,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:09,104 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304241e608758f32b49169718aa834f406f98 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304241e608758f32b49169718aa834f406f98 2023-04-24 09:00:09,114 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,114 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,114 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,115 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:09,116 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:09,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46dc67ed[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:09,116 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,135 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:09,136 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,136 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:09,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:09,136 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:09,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:09,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326869150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:09,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326869155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:09,184 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,184 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:09,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:09,186 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:09,186 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:09,186 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:09,186 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:09,186 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:09,254 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,254 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,254 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,254 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,255 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:09,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326869256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:09,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:09,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:09,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,265 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:09,265 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:09,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:09,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326869262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,268 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,268 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,365 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:09,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:09,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:09,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:09,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:09,367 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:09,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:09,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:09,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:09,368 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,368 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,379 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,379 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:09,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:09,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:09,380 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,380 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:09,380 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:09,380 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:09,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:09,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:09,391 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:09,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19015939[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:09,391 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,397 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,398 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:09,398 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,398 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:09,398 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:09,398 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:09,406 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:09,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,406 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,406 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:09,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:09,408 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:09,408 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:09,408 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:09,408 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:09,408 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:09,414 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,414 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,414 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,414 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,415 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:09,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:09,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,422 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,423 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:09,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:09,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,423 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:09,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:09,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326869462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:09,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326869469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,508 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=887, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/c43a161f95fe4124acf9d8b4779173c0 2023-04-24 09:00:09,523 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:09,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:09,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:09,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:09,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:09,526 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:09,526 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:09,526 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:09,526 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:09,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,527 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,537 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,537 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,537 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,538 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:09,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:09,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:09,538 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:09,538 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:09,538 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:09,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:09,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:09,539 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,539 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:09,539 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:09,539 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:09,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:09,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:09,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:09,547 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,547 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,548 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,548 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:09,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266bc3f9[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:09,549 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:09,549 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,556 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:09,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:09,556 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,556 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:09,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:09,564 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:09,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:09,566 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:09,566 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:09,566 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:09,566 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:09,566 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:09,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,573 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,574 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:09,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:09,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:09,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,615 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:09,615 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,615 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:09,715 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:09,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:09,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:09,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:09,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:09,718 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:09,718 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:09,718 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:09,718 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:09,718 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,718 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,737 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,737 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,737 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:09,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:09,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:09,738 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:09,739 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,739 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:09,739 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:09,739 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:09,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:09,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:09,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:09,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23da6d28[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:09,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,756 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:09,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:09,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:09,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:09,764 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:09,764 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,764 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,764 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:09,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:09,766 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:09,766 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:09,766 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:09,766 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:09,766 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:09,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:09,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326869768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,772 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,772 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,773 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,774 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:09,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 462 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326869775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:09,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:09,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:09,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,781 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:09,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,781 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:09,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,782 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,782 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,881 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:09,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:09,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:09,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:09,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:09,884 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:09,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:09,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:09,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:09,885 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,885 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,889 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:09,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:09,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:09,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:09,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:09,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:09,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:09,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:09,891 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,891 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:09,891 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:09,891 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,891 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:09,891 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:09,891 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,891 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:09,891 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:09,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,897 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:09,899 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:09,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c85b1[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:09,899 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,906 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:09,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:09,906 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:09,906 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:09,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:09,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/d37d33b803d8466c9e9e0707ef570f22 2023-04-24 09:00:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:09,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:09,918 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:09,918 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:09,918 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:09,918 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:09,918 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:09,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:09,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:09,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/567b992d927846128cb18dfb08ad905b 2023-04-24 09:00:09,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/c43a161f95fe4124acf9d8b4779173c0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c43a161f95fe4124acf9d8b4779173c0 2023-04-24 09:00:09,931 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,931 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,931 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:09,931 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:09,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:09,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:09,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c43a161f95fe4124acf9d8b4779173c0, entries=200, sequenceid=887, filesize=32.3 K 2023-04-24 09:00:09,931 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:09,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:09,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:09,931 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:09,931 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:09,932 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:09,931 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:09,932 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:09,932 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:09,932 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:09,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/d37d33b803d8466c9e9e0707ef570f22 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/d37d33b803d8466c9e9e0707ef570f22 2023-04-24 09:00:09,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/d37d33b803d8466c9e9e0707ef570f22, entries=150, sequenceid=887, filesize=11.9 K 2023-04-24 09:00:09,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/567b992d927846128cb18dfb08ad905b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/567b992d927846128cb18dfb08ad905b 2023-04-24 09:00:09,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/567b992d927846128cb18dfb08ad905b, entries=150, sequenceid=887, filesize=11.9 K 2023-04-24 09:00:09,939 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~238.36 KB/244080, currentSize=93.93 KB/96180 for d7e0c211118009ab258bf68c7108cae8 in 851ms, sequenceid=887, compaction requested=true 2023-04-24 09:00:09,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:09,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:09,939 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:09,939 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:09,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:09,939 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:09,940 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97708 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:09,940 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44436 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:09,940 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/A is initiating minor compaction (all files) 2023-04-24 09:00:09,940 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/B is initiating minor compaction (all files) 2023-04-24 09:00:09,940 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/A in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,940 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/B in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:09,940 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/892c0042acc741889a29c8d20e644d29, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/2a5412353e574139b0cc35d55533f7ba, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/41262fc129654adaa08b72491dc4325d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c43a161f95fe4124acf9d8b4779173c0] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=95.4 K 2023-04-24 09:00:09,940 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1c8af361aabe4b9fb5afec7505231310, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f51495e98a2442b3a67dcfd62152121f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/c91b2057420a465098ba9bab2d33b57b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/d37d33b803d8466c9e9e0707ef570f22] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=43.4 K 2023-04-24 09:00:09,941 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 892c0042acc741889a29c8d20e644d29, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1682326806011 2023-04-24 09:00:09,941 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 1c8af361aabe4b9fb5afec7505231310, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=597, earliestPutTs=1682326806011 2023-04-24 09:00:09,941 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 2a5412353e574139b0cc35d55533f7ba, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1682326806045 2023-04-24 09:00:09,941 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f51495e98a2442b3a67dcfd62152121f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1682326806045 2023-04-24 09:00:09,941 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 41262fc129654adaa08b72491dc4325d, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1682326808207 2023-04-24 09:00:09,941 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c91b2057420a465098ba9bab2d33b57b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1682326808207 2023-04-24 09:00:09,941 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c43a161f95fe4124acf9d8b4779173c0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682326809068 2023-04-24 09:00:09,943 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d37d33b803d8466c9e9e0707ef570f22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682326809068 2023-04-24 09:00:09,953 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#B#compaction#322 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:09,955 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#A#compaction#321 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:10,032 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:10,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:10,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:10,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:10,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:10,034 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:10,034 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:10,034 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:10,034 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:10,035 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:10,035 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:10,039 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:10,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:10,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:10,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:10,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:10,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:10,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:10,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:10,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:10,041 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:10,041 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:10,041 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:10,041 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:10,041 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:10,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:10,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. started... 2023-04-24 09:00:10,041 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-24 09:00:10,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:10,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:10,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:10,042 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:10,042 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:10,042 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:10,063 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424853b74de4bdb4111869feb2e8ba9cf4a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424853b74de4bdb4111869feb2e8ba9cf4a 2023-04-24 09:00:10,078 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=904, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/0bd9f57adf35469bb33a2f4710ec7154 2023-04-24 09:00:10,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:10,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. as already flushing 2023-04-24 09:00:10,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:10,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 497 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326870305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:10,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326870316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,364 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/6877a2c3ea274442b116b99a7a94c94f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/6877a2c3ea274442b116b99a7a94c94f 2023-04-24 09:00:10,368 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/B of d7e0c211118009ab258bf68c7108cae8 into 6877a2c3ea274442b116b99a7a94c94f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:10,368 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:10,368 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/B, priority=12, startTime=1682326809939; duration=0sec 2023-04-24 09:00:10,368 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:10,368 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:10,369 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34728 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:10,369 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/C is initiating minor compaction (all files) 2023-04-24 09:00:10,369 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/C in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:10,369 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/768c0d4233ad4660af5e164b10182612, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/afd9a45e58704baeb6d712c06125346c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/567b992d927846128cb18dfb08ad905b] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=33.9 K 2023-04-24 09:00:10,369 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 768c0d4233ad4660af5e164b10182612, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1682326806013 2023-04-24 09:00:10,369 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting afd9a45e58704baeb6d712c06125346c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1682326808207 2023-04-24 09:00:10,369 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 567b992d927846128cb18dfb08ad905b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682326809068 2023-04-24 09:00:10,374 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#C#compaction#325 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:10,379 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/fa0c106a71e3477699ec55bf516d086e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/fa0c106a71e3477699ec55bf516d086e 2023-04-24 09:00:10,382 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/68f2e184ed714254ba409d2f79a85694 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/68f2e184ed714254ba409d2f79a85694 2023-04-24 09:00:10,385 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/A of d7e0c211118009ab258bf68c7108cae8 into fa0c106a71e3477699ec55bf516d086e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:10,385 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:10,385 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/A, priority=12, startTime=1682326809939; duration=0sec 2023-04-24 09:00:10,385 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:10,388 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/C of d7e0c211118009ab258bf68c7108cae8 into 68f2e184ed714254ba409d2f79a85694(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:10,388 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:10,388 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/C, priority=13, startTime=1682326809939; duration=0sec 2023-04-24 09:00:10,388 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:10,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:10,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 499 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326870420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:10,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326870427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,434 DEBUG [Thread-2002] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52829d32 to 127.0.0.1:63978 2023-04-24 09:00:10,434 DEBUG [Thread-2002] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:10,435 DEBUG [Thread-2006] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3558849b to 127.0.0.1:63978 2023-04-24 09:00:10,435 DEBUG [Thread-2006] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:10,439 DEBUG [Thread-2004] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x516a95ee to 127.0.0.1:63978 2023-04-24 09:00:10,439 DEBUG [Thread-2004] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:10,440 DEBUG [Thread-2000] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17385c96 to 127.0.0.1:63978 2023-04-24 09:00:10,440 DEBUG [Thread-2000] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:10,441 DEBUG [Thread-2008] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24d104d6 to 127.0.0.1:63978 2023-04-24 09:00:10,441 DEBUG [Thread-2008] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:10,502 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/bb42f005b28341faa3e12f4bb558d4ae 2023-04-24 09:00:10,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:10,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 501 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326870627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:10,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326870635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,922 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=904 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/d01976d755e34bb0ac811c3db6f98c9c 2023-04-24 09:00:10,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:10,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 503 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52690 deadline: 1682326870930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,932 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/0bd9f57adf35469bb33a2f4710ec7154 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0bd9f57adf35469bb33a2f4710ec7154 2023-04-24 09:00:10,937 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0bd9f57adf35469bb33a2f4710ec7154, entries=150, sequenceid=904, filesize=25.4 K 2023-04-24 09:00:10,938 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/bb42f005b28341faa3e12f4bb558d4ae as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/bb42f005b28341faa3e12f4bb558d4ae 2023-04-24 09:00:10,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:10,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:52680 deadline: 1682326870939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,943 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/bb42f005b28341faa3e12f4bb558d4ae, entries=150, sequenceid=904, filesize=11.9 K 2023-04-24 09:00:10,944 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/d01976d755e34bb0ac811c3db6f98c9c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d01976d755e34bb0ac811c3db6f98c9c 2023-04-24 09:00:10,953 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d01976d755e34bb0ac811c3db6f98c9c, entries=150, sequenceid=904, filesize=11.9 K 2023-04-24 09:00:10,954 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d7e0c211118009ab258bf68c7108cae8 in 913ms, sequenceid=904, compaction requested=false 2023-04-24 09:00:10,954 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:10,954 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:10,954 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:10,954 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:10,954 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,954 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:10,954 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:10,990 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,990 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:10,991 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:10,991 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:10,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:10,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c7ccca[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:10,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:10,996 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:11,006 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:11,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:11,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:11,007 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:11,007 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:11,007 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:11,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:11,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:11,014 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:11,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:11,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:11,019 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:11,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:11,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:11,019 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:11,019 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:11,029 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:11,029 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:11,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:11,029 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:11,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:11,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,033 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,066 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,067 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:11,067 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:11,067 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:11,067 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:11,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:11,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:11,066 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:11,067 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:11,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:11,068 DEBUG [Thread-1999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:11,068 DEBUG [Thread-1999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:11,067 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:11,068 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:11,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:11,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:11,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:11,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:11,168 DEBUG [Thread-1999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:11,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:11,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:11,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-24 09:00:11,437 DEBUG [Thread-1991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c70a497 to 127.0.0.1:63978 2023-04-24 09:00:11,437 DEBUG [Thread-1991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:11,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:11,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:11,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:11,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:11,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:11,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:11,443 DEBUG [Thread-1997] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4de67ff9 to 127.0.0.1:63978 2023-04-24 09:00:11,443 DEBUG [Thread-1997] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:11,854 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424a41bbac082e348caa3baa6e16620c0f5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a41bbac082e348caa3baa6e16620c0f5 2023-04-24 09:00:12,261 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=923, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/bef0225515a0457990f03c8cdca51a06 2023-04-24 09:00:12,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/a0876106d16f41088d5b6b1ce697ef19 2023-04-24 09:00:13,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/fd581fec05d94e349228f5123bdc0779 2023-04-24 09:00:13,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/bef0225515a0457990f03c8cdca51a06 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/bef0225515a0457990f03c8cdca51a06 2023-04-24 09:00:13,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/bef0225515a0457990f03c8cdca51a06, entries=150, sequenceid=923, filesize=25.4 K 2023-04-24 09:00:13,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/a0876106d16f41088d5b6b1ce697ef19 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a0876106d16f41088d5b6b1ce697ef19 2023-04-24 09:00:13,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a0876106d16f41088d5b6b1ce697ef19, entries=150, sequenceid=923, filesize=11.9 K 2023-04-24 09:00:13,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/fd581fec05d94e349228f5123bdc0779 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fd581fec05d94e349228f5123bdc0779 2023-04-24 09:00:13,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fd581fec05d94e349228f5123bdc0779, entries=150, sequenceid=923, filesize=11.9 K 2023-04-24 09:00:13,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=6.71 KB/6870 for d7e0c211118009ab258bf68c7108cae8 in 1669ms, sequenceid=923, compaction requested=true 2023-04-24 09:00:13,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:13,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:13,106 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:13,106 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:13,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:13,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78854 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/A is initiating minor compaction (all files) 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/B is initiating minor compaction (all files) 2023-04-24 09:00:13,107 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/A in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:13,107 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/B in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:13,107 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/fa0c106a71e3477699ec55bf516d086e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0bd9f57adf35469bb33a2f4710ec7154, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/bef0225515a0457990f03c8cdca51a06] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=77.0 K 2023-04-24 09:00:13,107 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/6877a2c3ea274442b116b99a7a94c94f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/bb42f005b28341faa3e12f4bb558d4ae, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a0876106d16f41088d5b6b1ce697ef19] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=36.4 K 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting fa0c106a71e3477699ec55bf516d086e, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682326809068 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 6877a2c3ea274442b116b99a7a94c94f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682326809068 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 0bd9f57adf35469bb33a2f4710ec7154, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1682326809136 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting bb42f005b28341faa3e12f4bb558d4ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1682326809136 2023-04-24 09:00:13,107 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting bef0225515a0457990f03c8cdca51a06, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1682326810304 2023-04-24 09:00:13,108 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a0876106d16f41088d5b6b1ce697ef19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1682326810304 2023-04-24 09:00:13,116 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#B#compaction#330 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:13,126 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#A#compaction#331 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:13,538 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/119c892b64a04116b9ab8bf3c9d97603 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/119c892b64a04116b9ab8bf3c9d97603 2023-04-24 09:00:13,545 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/B of d7e0c211118009ab258bf68c7108cae8 into 119c892b64a04116b9ab8bf3c9d97603(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:13,545 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:13,545 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/B, priority=13, startTime=1682326813106; duration=0sec 2023-04-24 09:00:13,545 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:13,545 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:13,546 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:13,546 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d7e0c211118009ab258bf68c7108cae8/C is initiating minor compaction (all files) 2023-04-24 09:00:13,546 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d7e0c211118009ab258bf68c7108cae8/C in TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:13,546 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/68f2e184ed714254ba409d2f79a85694, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d01976d755e34bb0ac811c3db6f98c9c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fd581fec05d94e349228f5123bdc0779] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp, totalSize=36.4 K 2023-04-24 09:00:13,547 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 68f2e184ed714254ba409d2f79a85694, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1682326809068 2023-04-24 09:00:13,547 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d01976d755e34bb0ac811c3db6f98c9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=904, earliestPutTs=1682326809136 2023-04-24 09:00:13,548 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting fd581fec05d94e349228f5123bdc0779, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1682326810304 2023-04-24 09:00:13,550 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/1e337fc3f6c848a7b86cb5105cdcbce7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1e337fc3f6c848a7b86cb5105cdcbce7 2023-04-24 09:00:13,554 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d7e0c211118009ab258bf68c7108cae8#C#compaction#332 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:13,554 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/A of d7e0c211118009ab258bf68c7108cae8 into 1e337fc3f6c848a7b86cb5105cdcbce7(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:13,554 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:13,554 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/A, priority=13, startTime=1682326813106; duration=0sec 2023-04-24 09:00:13,555 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:13,971 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/83d11942ee72466f8b6ffc6a7fb44a29 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/83d11942ee72466f8b6ffc6a7fb44a29 2023-04-24 09:00:13,977 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d7e0c211118009ab258bf68c7108cae8/C of d7e0c211118009ab258bf68c7108cae8 into 83d11942ee72466f8b6ffc6a7fb44a29(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:13,977 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:13,977 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8., storeName=d7e0c211118009ab258bf68c7108cae8/C, priority=13, startTime=1682326813106; duration=0sec 2023-04-24 09:00:13,977 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:15,272 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/3fcc0b508b1646f09533d4b48b6641bc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c8f5cb9622854b6ab90fc5c1e65b4f0c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/499c6189073f4812aacb7d50dc7bf790, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/ba93366242b24b7da5b9b5c086ab9423, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e7a7d9894b1e4a5fbfd2a041e79203b0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1cebbf014be4488aac7726d4a7408814, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/9c925dfc7b004f4f9608182e34438852, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cf211ad810d14cb9bd088049ca6b4415, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/11563e8f6057410098ae7f83d9d06ae3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/a0ef0867773d4de990f6a025a6946308, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/4648447d8f2c4475b295e27d4ad76147, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e0dabe0251794b2eaba5d646093a4055, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/142752f799ab4675a1366292c89376f5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/17fd2805c29f413db2b42577000bfb14, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cbe877dbc4ed4e54a9cb3960ec7f0f9e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/78a99ac0e9d5432eb39cc7d1751131a4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0c0f65a576934c82a6c7c54f216c950d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/892c0042acc741889a29c8d20e644d29, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/2a5412353e574139b0cc35d55533f7ba, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/41262fc129654adaa08b72491dc4325d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c43a161f95fe4124acf9d8b4779173c0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/fa0c106a71e3477699ec55bf516d086e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0bd9f57adf35469bb33a2f4710ec7154, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/bef0225515a0457990f03c8cdca51a06] to archive 2023-04-24 09:00:15,272 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:00:15,274 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/3fcc0b508b1646f09533d4b48b6641bc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/3fcc0b508b1646f09533d4b48b6641bc 2023-04-24 09:00:15,275 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c8f5cb9622854b6ab90fc5c1e65b4f0c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c8f5cb9622854b6ab90fc5c1e65b4f0c 2023-04-24 09:00:15,276 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/499c6189073f4812aacb7d50dc7bf790 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/499c6189073f4812aacb7d50dc7bf790 2023-04-24 09:00:15,277 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/ba93366242b24b7da5b9b5c086ab9423 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/ba93366242b24b7da5b9b5c086ab9423 2023-04-24 09:00:15,278 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e7a7d9894b1e4a5fbfd2a041e79203b0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e7a7d9894b1e4a5fbfd2a041e79203b0 2023-04-24 09:00:15,279 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1cebbf014be4488aac7726d4a7408814 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1cebbf014be4488aac7726d4a7408814 2023-04-24 09:00:15,280 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/9c925dfc7b004f4f9608182e34438852 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/9c925dfc7b004f4f9608182e34438852 2023-04-24 09:00:15,281 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cf211ad810d14cb9bd088049ca6b4415 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cf211ad810d14cb9bd088049ca6b4415 2023-04-24 09:00:15,282 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/11563e8f6057410098ae7f83d9d06ae3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/11563e8f6057410098ae7f83d9d06ae3 2023-04-24 09:00:15,283 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/a0ef0867773d4de990f6a025a6946308 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/a0ef0867773d4de990f6a025a6946308 2023-04-24 09:00:15,285 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/4648447d8f2c4475b295e27d4ad76147 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/4648447d8f2c4475b295e27d4ad76147 2023-04-24 09:00:15,286 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e0dabe0251794b2eaba5d646093a4055 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/e0dabe0251794b2eaba5d646093a4055 2023-04-24 09:00:15,287 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/142752f799ab4675a1366292c89376f5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/142752f799ab4675a1366292c89376f5 2023-04-24 09:00:15,287 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/17fd2805c29f413db2b42577000bfb14 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/17fd2805c29f413db2b42577000bfb14 2023-04-24 09:00:15,289 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cbe877dbc4ed4e54a9cb3960ec7f0f9e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/cbe877dbc4ed4e54a9cb3960ec7f0f9e 2023-04-24 09:00:15,290 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/78a99ac0e9d5432eb39cc7d1751131a4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/78a99ac0e9d5432eb39cc7d1751131a4 2023-04-24 09:00:15,291 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0c0f65a576934c82a6c7c54f216c950d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0c0f65a576934c82a6c7c54f216c950d 2023-04-24 09:00:15,292 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/892c0042acc741889a29c8d20e644d29 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/892c0042acc741889a29c8d20e644d29 2023-04-24 09:00:15,292 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/2a5412353e574139b0cc35d55533f7ba to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/2a5412353e574139b0cc35d55533f7ba 2023-04-24 09:00:15,293 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/41262fc129654adaa08b72491dc4325d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/41262fc129654adaa08b72491dc4325d 2023-04-24 09:00:15,294 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c43a161f95fe4124acf9d8b4779173c0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/c43a161f95fe4124acf9d8b4779173c0 2023-04-24 09:00:15,295 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/fa0c106a71e3477699ec55bf516d086e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/fa0c106a71e3477699ec55bf516d086e 2023-04-24 09:00:15,295 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0bd9f57adf35469bb33a2f4710ec7154 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/0bd9f57adf35469bb33a2f4710ec7154 2023-04-24 09:00:15,296 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/bef0225515a0457990f03c8cdca51a06 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/bef0225515a0457990f03c8cdca51a06 2023-04-24 09:00:15,302 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/5dd2a512fb92405b9f3aa3bc393906c3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f9fa879f43dc4846a2569c6264f92cb9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a957da97655f47e4a24accb702e0a2e9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/996046a403804cf99a6eacc7a19861a7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/7be79b168dbc41279987b0f9599ce9b3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/cded3c10c5cc441c90411d71ddca38e6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/feea3472a815434587ff1300085fec11, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/312ccc8776f04a55aeb90a2d840e317b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/55557db42b534af3a84e1f2066c55168, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2b7556f4e0ea46cd8d29e23011ed7792, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f566829c327e403d81fe4c9df734fac0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/9bcce601e144425e83980681bcf2a32f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2ab693a4b98d4deab30f86cae5f70703, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e84e9cd62db54fb6bce3da99a09f71af, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1fc24fde52f64411ab3713a5e71eba2f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a6bc518a510040dabf6d9b69860712cc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1c8af361aabe4b9fb5afec7505231310, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/ea7aa5fa7d6b4cef96c13dad150b472c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f51495e98a2442b3a67dcfd62152121f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/c91b2057420a465098ba9bab2d33b57b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/6877a2c3ea274442b116b99a7a94c94f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/d37d33b803d8466c9e9e0707ef570f22, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/bb42f005b28341faa3e12f4bb558d4ae, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a0876106d16f41088d5b6b1ce697ef19] to archive 2023-04-24 09:00:15,303 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:00:15,304 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/5dd2a512fb92405b9f3aa3bc393906c3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/5dd2a512fb92405b9f3aa3bc393906c3 2023-04-24 09:00:15,305 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f9fa879f43dc4846a2569c6264f92cb9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f9fa879f43dc4846a2569c6264f92cb9 2023-04-24 09:00:15,306 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a957da97655f47e4a24accb702e0a2e9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a957da97655f47e4a24accb702e0a2e9 2023-04-24 09:00:15,307 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/996046a403804cf99a6eacc7a19861a7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/996046a403804cf99a6eacc7a19861a7 2023-04-24 09:00:15,307 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/7be79b168dbc41279987b0f9599ce9b3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/7be79b168dbc41279987b0f9599ce9b3 2023-04-24 09:00:15,308 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/cded3c10c5cc441c90411d71ddca38e6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/cded3c10c5cc441c90411d71ddca38e6 2023-04-24 09:00:15,309 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/feea3472a815434587ff1300085fec11 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/feea3472a815434587ff1300085fec11 2023-04-24 09:00:15,310 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/312ccc8776f04a55aeb90a2d840e317b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/312ccc8776f04a55aeb90a2d840e317b 2023-04-24 09:00:15,310 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/55557db42b534af3a84e1f2066c55168 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/55557db42b534af3a84e1f2066c55168 2023-04-24 09:00:15,311 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2b7556f4e0ea46cd8d29e23011ed7792 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2b7556f4e0ea46cd8d29e23011ed7792 2023-04-24 09:00:15,312 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f566829c327e403d81fe4c9df734fac0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f566829c327e403d81fe4c9df734fac0 2023-04-24 09:00:15,313 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/9bcce601e144425e83980681bcf2a32f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/9bcce601e144425e83980681bcf2a32f 2023-04-24 09:00:15,314 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2ab693a4b98d4deab30f86cae5f70703 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/2ab693a4b98d4deab30f86cae5f70703 2023-04-24 09:00:15,315 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e84e9cd62db54fb6bce3da99a09f71af to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e84e9cd62db54fb6bce3da99a09f71af 2023-04-24 09:00:15,315 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1fc24fde52f64411ab3713a5e71eba2f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1fc24fde52f64411ab3713a5e71eba2f 2023-04-24 09:00:15,316 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a6bc518a510040dabf6d9b69860712cc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a6bc518a510040dabf6d9b69860712cc 2023-04-24 09:00:15,317 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1c8af361aabe4b9fb5afec7505231310 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/1c8af361aabe4b9fb5afec7505231310 2023-04-24 09:00:15,318 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/ea7aa5fa7d6b4cef96c13dad150b472c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/ea7aa5fa7d6b4cef96c13dad150b472c 2023-04-24 09:00:15,319 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f51495e98a2442b3a67dcfd62152121f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/f51495e98a2442b3a67dcfd62152121f 2023-04-24 09:00:15,320 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/c91b2057420a465098ba9bab2d33b57b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/c91b2057420a465098ba9bab2d33b57b 2023-04-24 09:00:15,321 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/6877a2c3ea274442b116b99a7a94c94f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/6877a2c3ea274442b116b99a7a94c94f 2023-04-24 09:00:15,322 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/d37d33b803d8466c9e9e0707ef570f22 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/d37d33b803d8466c9e9e0707ef570f22 2023-04-24 09:00:15,323 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/bb42f005b28341faa3e12f4bb558d4ae to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/bb42f005b28341faa3e12f4bb558d4ae 2023-04-24 09:00:15,324 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a0876106d16f41088d5b6b1ce697ef19 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/a0876106d16f41088d5b6b1ce697ef19 2023-04-24 09:00:15,330 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/da591db4a47d4fc8b85d898f6d7c4b0d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/11a5a8789b124f3ca0c1a5ad51c74ff0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/78b8de05792c42fdb085d619ed4c4217, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a69a130e5273421eaf51ad28b13cbe17, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/1128bf44fbfc41dba5cf6bc708a18eeb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/0cd858b2ed124ace870e50a41fa554aa, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/372e0a8ae1ab4dd595212eae5d3a70de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/540671d3bfe347cdb676c4872007fdd3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/170fa559d7584fce847edce39151c06d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/e8f92b0d2b4848bf82c9de3f8bdbfeb9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/4f7d906d1a3249aaa83bde50fe29c27e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fdf9f56fe91d4218b65f520de6ab5342, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d4b30315d7704d4984e4cb0280d665d4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/3238effc6fb348429f17c9009f122103, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a9d00784f5304fd7b01d1bb460f6ea13, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/38fa2091a79740bb83a175adbdebb97d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/232bfa736eb144baa18c1d047f8e4629, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/768c0d4233ad4660af5e164b10182612, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/825d4c2c81bc492a8d1e24e417c37ca8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/afd9a45e58704baeb6d712c06125346c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/68f2e184ed714254ba409d2f79a85694, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/567b992d927846128cb18dfb08ad905b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d01976d755e34bb0ac811c3db6f98c9c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fd581fec05d94e349228f5123bdc0779] to archive 2023-04-24 09:00:15,331 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:00:15,333 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/da591db4a47d4fc8b85d898f6d7c4b0d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/da591db4a47d4fc8b85d898f6d7c4b0d 2023-04-24 09:00:15,334 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/11a5a8789b124f3ca0c1a5ad51c74ff0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/11a5a8789b124f3ca0c1a5ad51c74ff0 2023-04-24 09:00:15,336 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/78b8de05792c42fdb085d619ed4c4217 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/78b8de05792c42fdb085d619ed4c4217 2023-04-24 09:00:15,337 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a69a130e5273421eaf51ad28b13cbe17 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a69a130e5273421eaf51ad28b13cbe17 2023-04-24 09:00:15,338 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/1128bf44fbfc41dba5cf6bc708a18eeb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/1128bf44fbfc41dba5cf6bc708a18eeb 2023-04-24 09:00:15,340 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/0cd858b2ed124ace870e50a41fa554aa to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/0cd858b2ed124ace870e50a41fa554aa 2023-04-24 09:00:15,341 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/372e0a8ae1ab4dd595212eae5d3a70de to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/372e0a8ae1ab4dd595212eae5d3a70de 2023-04-24 09:00:15,342 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/540671d3bfe347cdb676c4872007fdd3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/540671d3bfe347cdb676c4872007fdd3 2023-04-24 09:00:15,343 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/170fa559d7584fce847edce39151c06d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/170fa559d7584fce847edce39151c06d 2023-04-24 09:00:15,344 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/e8f92b0d2b4848bf82c9de3f8bdbfeb9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/e8f92b0d2b4848bf82c9de3f8bdbfeb9 2023-04-24 09:00:15,345 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/4f7d906d1a3249aaa83bde50fe29c27e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/4f7d906d1a3249aaa83bde50fe29c27e 2023-04-24 09:00:15,346 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fdf9f56fe91d4218b65f520de6ab5342 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fdf9f56fe91d4218b65f520de6ab5342 2023-04-24 09:00:15,347 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d4b30315d7704d4984e4cb0280d665d4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d4b30315d7704d4984e4cb0280d665d4 2023-04-24 09:00:15,347 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/3238effc6fb348429f17c9009f122103 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/3238effc6fb348429f17c9009f122103 2023-04-24 09:00:15,348 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a9d00784f5304fd7b01d1bb460f6ea13 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/a9d00784f5304fd7b01d1bb460f6ea13 2023-04-24 09:00:15,349 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/38fa2091a79740bb83a175adbdebb97d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/38fa2091a79740bb83a175adbdebb97d 2023-04-24 09:00:15,350 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/232bfa736eb144baa18c1d047f8e4629 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/232bfa736eb144baa18c1d047f8e4629 2023-04-24 09:00:15,351 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/768c0d4233ad4660af5e164b10182612 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/768c0d4233ad4660af5e164b10182612 2023-04-24 09:00:15,352 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/825d4c2c81bc492a8d1e24e417c37ca8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/825d4c2c81bc492a8d1e24e417c37ca8 2023-04-24 09:00:15,353 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/afd9a45e58704baeb6d712c06125346c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/afd9a45e58704baeb6d712c06125346c 2023-04-24 09:00:15,354 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/68f2e184ed714254ba409d2f79a85694 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/68f2e184ed714254ba409d2f79a85694 2023-04-24 09:00:15,355 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/567b992d927846128cb18dfb08ad905b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/567b992d927846128cb18dfb08ad905b 2023-04-24 09:00:15,355 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d01976d755e34bb0ac811c3db6f98c9c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d01976d755e34bb0ac811c3db6f98c9c 2023-04-24 09:00:15,356 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fd581fec05d94e349228f5123bdc0779 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/fd581fec05d94e349228f5123bdc0779 2023-04-24 09:00:16,129 DEBUG [Thread-1995] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61483370 to 127.0.0.1:63978 2023-04-24 09:00:16,129 DEBUG [Thread-1995] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:16,147 DEBUG [Thread-1989] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f54fc1a to 127.0.0.1:63978 2023-04-24 09:00:16,147 DEBUG [Thread-1989] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:16,158 DEBUG [Thread-1993] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x179d9850 to 127.0.0.1:63978 2023-04-24 09:00:16,158 DEBUG [Thread-1993] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:16,159 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 09:00:16,159 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-24 09:00:16,159 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 404 2023-04-24 09:00:16,159 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-04-24 09:00:16,160 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-24 09:00:16,160 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 370 2023-04-24 09:00:16,160 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 09:00:16,160 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 09:00:16,160 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 1974 2023-04-24 09:00:16,160 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 5917 rows 2023-04-24 09:00:16,160 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 1984 2023-04-24 09:00:16,161 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 5944 rows 2023-04-24 09:00:16,161 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 1991 2023-04-24 09:00:16,161 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 5961 rows 2023-04-24 09:00:16,161 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 1981 2023-04-24 09:00:16,161 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 5943 rows 2023-04-24 09:00:16,161 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 1977 2023-04-24 09:00:16,161 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 5931 rows 2023-04-24 09:00:16,161 INFO [Listener at localhost.localdomain/44291] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 09:00:16,162 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x727ea8af to 127.0.0.1:63978 2023-04-24 09:00:16,162 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:16,163 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 09:00:16,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-24 09:00:16,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:16,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 09:00:16,168 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326816168"}]},"ts":"1682326816168"} 2023-04-24 09:00:16,168 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 09:00:16,197 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 09:00:16,199 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, UNASSIGN}] 2023-04-24 09:00:16,200 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, UNASSIGN 2023-04-24 09:00:16,201 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=d7e0c211118009ab258bf68c7108cae8, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:16,201 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326816201"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326816201"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326816201"}]},"ts":"1682326816201"} 2023-04-24 09:00:16,203 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 09:00:16,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 09:00:16,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:16,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing d7e0c211118009ab258bf68c7108cae8, disabling compactions & flushes 2023-04-24 09:00:16,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:16,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:16,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. after waiting 0 ms 2023-04-24 09:00:16,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:16,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing d7e0c211118009ab258bf68c7108cae8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 09:00:16,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=A 2023-04-24 09:00:16,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:16,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=B 2023-04-24 09:00:16,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:16,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d7e0c211118009ab258bf68c7108cae8, store=C 2023-04-24 09:00:16,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:16,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 09:00:16,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 09:00:16,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424b59cb820e0114181982a3f99c6bf38ff to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b59cb820e0114181982a3f99c6bf38ff 2023-04-24 09:00:17,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=933, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/360e7fcef6564330bbba99421e6f1566 2023-04-24 09:00:17,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 09:00:17,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=933 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/e0b839a811ff4792b12137b246146563 2023-04-24 09:00:18,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=933 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/d30ff33d683e4bbfb26d821778a55db1 2023-04-24 09:00:18,024 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/A/360e7fcef6564330bbba99421e6f1566 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/360e7fcef6564330bbba99421e6f1566 2023-04-24 09:00:18,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/360e7fcef6564330bbba99421e6f1566, entries=100, sequenceid=933, filesize=18.6 K 2023-04-24 09:00:18,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/B/e0b839a811ff4792b12137b246146563 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e0b839a811ff4792b12137b246146563 2023-04-24 09:00:18,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e0b839a811ff4792b12137b246146563, entries=100, sequenceid=933, filesize=9.5 K 2023-04-24 09:00:18,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/.tmp/C/d30ff33d683e4bbfb26d821778a55db1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d30ff33d683e4bbfb26d821778a55db1 2023-04-24 09:00:18,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d30ff33d683e4bbfb26d821778a55db1, entries=100, sequenceid=933, filesize=9.5 K 2023-04-24 09:00:18,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d7e0c211118009ab258bf68c7108cae8 in 1689ms, sequenceid=933, compaction requested=false 2023-04-24 09:00:18,057 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/recovered.edits/936.seqid, newMaxSeqId=936, maxSeqId=4 2023-04-24 09:00:18,058 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8. 2023-04-24 09:00:18,058 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for d7e0c211118009ab258bf68c7108cae8: 2023-04-24 09:00:18,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:18,060 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=d7e0c211118009ab258bf68c7108cae8, regionState=CLOSED 2023-04-24 09:00:18,060 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326818060"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326818060"}]},"ts":"1682326818060"} 2023-04-24 09:00:18,062 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-24 09:00:18,062 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure d7e0c211118009ab258bf68c7108cae8, server=jenkins-hbase16.apache.org,42837,1682326693550 in 1.8580 sec 2023-04-24 09:00:18,064 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-24 09:00:18,064 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d7e0c211118009ab258bf68c7108cae8, UNASSIGN in 1.8630 sec 2023-04-24 09:00:18,064 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326818064"}]},"ts":"1682326818064"} 2023-04-24 09:00:18,065 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 09:00:18,097 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 09:00:18,100 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9340 sec 2023-04-24 09:00:18,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-24 09:00:18,277 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-24 09:00:18,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-24 09:00:18,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:18,284 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:18,286 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:18,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-24 09:00:18,292 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:18,297 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/recovered.edits] 2023-04-24 09:00:18,303 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1e337fc3f6c848a7b86cb5105cdcbce7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/1e337fc3f6c848a7b86cb5105cdcbce7 2023-04-24 09:00:18,306 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/360e7fcef6564330bbba99421e6f1566 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/A/360e7fcef6564330bbba99421e6f1566 2023-04-24 09:00:18,309 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/119c892b64a04116b9ab8bf3c9d97603 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/119c892b64a04116b9ab8bf3c9d97603 2023-04-24 09:00:18,311 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e0b839a811ff4792b12137b246146563 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/B/e0b839a811ff4792b12137b246146563 2023-04-24 09:00:18,314 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/83d11942ee72466f8b6ffc6a7fb44a29 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/83d11942ee72466f8b6ffc6a7fb44a29 2023-04-24 09:00:18,315 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d30ff33d683e4bbfb26d821778a55db1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/C/d30ff33d683e4bbfb26d821778a55db1 2023-04-24 09:00:18,318 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/recovered.edits/936.seqid to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8/recovered.edits/936.seqid 2023-04-24 09:00:18,318 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d7e0c211118009ab258bf68c7108cae8 2023-04-24 09:00:18,318 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 09:00:18,319 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 09:00:18,319 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-24 09:00:18,323 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240032fddb90b84865a8f07d1427adc8d6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240032fddb90b84865a8f07d1427adc8d6 2023-04-24 09:00:18,325 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042407d010f3005e41079aaf55a6064d0d94 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042407d010f3005e41079aaf55a6064d0d94 2023-04-24 09:00:18,327 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424171e2f35d6964eb39c78f5157ef57367 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424171e2f35d6964eb39c78f5157ef57367 2023-04-24 09:00:18,329 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304241e47e3bfad894231a70ffe4404f6ec9b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304241e47e3bfad894231a70ffe4404f6ec9b 2023-04-24 09:00:18,331 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304241e608758f32b49169718aa834f406f98 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304241e608758f32b49169718aa834f406f98 2023-04-24 09:00:18,332 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242f9f4b7355604e09b3ac6ab6da277426 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304242f9f4b7355604e09b3ac6ab6da277426 2023-04-24 09:00:18,334 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243321957880194c32833b1e02d8fdcedd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243321957880194c32833b1e02d8fdcedd 2023-04-24 09:00:18,336 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243a5314084a084fb4b10e6acbeb879509 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304243a5314084a084fb4b10e6acbeb879509 2023-04-24 09:00:18,338 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042442676c35a5a74124bbb4c192ef612d75 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042442676c35a5a74124bbb4c192ef612d75 2023-04-24 09:00:18,340 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042448b45d7afdbd4ece965ad21800f921f2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042448b45d7afdbd4ece965ad21800f921f2 2023-04-24 09:00:18,342 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424532358a2ebcc496789159fab863ae7ed to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424532358a2ebcc496789159fab863ae7ed 2023-04-24 09:00:18,345 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042458bb6680fc1b4ff4b881dac419135567 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042458bb6680fc1b4ff4b881dac419135567 2023-04-24 09:00:18,347 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042462dafb7c14044dd3bd5aa6a01381313a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042462dafb7c14044dd3bd5aa6a01381313a 2023-04-24 09:00:18,350 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246d4ce7ff24b84780a1066baf93e773d8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246d4ce7ff24b84780a1066baf93e773d8 2023-04-24 09:00:18,352 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246df696cd3de84133a091b015e5bc2c17 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304246df696cd3de84133a091b015e5bc2c17 2023-04-24 09:00:18,354 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424853b74de4bdb4111869feb2e8ba9cf4a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424853b74de4bdb4111869feb2e8ba9cf4a 2023-04-24 09:00:18,355 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a41bbac082e348caa3baa6e16620c0f5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424a41bbac082e348caa3baa6e16620c0f5 2023-04-24 09:00:18,357 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b59cb820e0114181982a3f99c6bf38ff to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424b59cb820e0114181982a3f99c6bf38ff 2023-04-24 09:00:18,358 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e4f3f4cb8e774ed5b04fa0d5853f1c9c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e4f3f4cb8e774ed5b04fa0d5853f1c9c 2023-04-24 09:00:18,359 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f6ceffddc967467493d48a0998f353d2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f6ceffddc967467493d48a0998f353d2 2023-04-24 09:00:18,359 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 09:00:18,361 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:18,362 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 09:00:18,363 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 09:00:18,364 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:18,364 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 09:00:18,364 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682326818364"}]},"ts":"9223372036854775807"} 2023-04-24 09:00:18,365 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 09:00:18,365 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d7e0c211118009ab258bf68c7108cae8, NAME => 'TestAcidGuarantees,,1682326787183.d7e0c211118009ab258bf68c7108cae8.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 09:00:18,366 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 09:00:18,366 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682326818366"}]},"ts":"9223372036854775807"} 2023-04-24 09:00:18,367 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 09:00:18,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-24 09:00:18,407 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:18,409 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 128 msec 2023-04-24 09:00:18,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-24 09:00:18,590 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-24 09:00:18,607 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 228) - Thread LEAK? -, OpenFileDescriptor=544 (was 531) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=305 (was 328), ProcessCount=165 (was 168), AvailableMemoryMB=8319 (was 8328) 2023-04-24 09:00:18,616 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=544, MaxFileDescriptor=60000, SystemLoadAverage=305, ProcessCount=165, AvailableMemoryMB=8318 2023-04-24 09:00:18,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 09:00:18,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 09:00:18,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:18,620 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 09:00:18,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-24 09:00:18,621 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 09:00:18,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 09:00:18,622 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:18,622 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990 empty. 2023-04-24 09:00:18,623 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:18,623 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 09:00:18,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 09:00:18,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 09:00:19,034 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 09:00:19,038 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3695ff097ea6e04d645670eb37341990, NAME => 'TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp 2023-04-24 09:00:19,046 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 09:00:19,046 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3695ff097ea6e04d645670eb37341990, disabling compactions & flushes 2023-04-24 09:00:19,046 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,046 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,046 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. after waiting 0 ms 2023-04-24 09:00:19,046 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,046 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,046 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:19,048 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 09:00:19,049 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326819049"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326819049"}]},"ts":"1682326819049"} 2023-04-24 09:00:19,050 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 09:00:19,051 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 09:00:19,051 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326819051"}]},"ts":"1682326819051"} 2023-04-24 09:00:19,051 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 09:00:19,097 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3695ff097ea6e04d645670eb37341990, ASSIGN}] 2023-04-24 09:00:19,099 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3695ff097ea6e04d645670eb37341990, ASSIGN 2023-04-24 09:00:19,100 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3695ff097ea6e04d645670eb37341990, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=false 2023-04-24 09:00:19,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 09:00:19,252 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=3695ff097ea6e04d645670eb37341990, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,252 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326819252"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326819252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326819252"}]},"ts":"1682326819252"} 2023-04-24 09:00:19,253 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 09:00:19,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3695ff097ea6e04d645670eb37341990, NAME => 'TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.', STARTKEY => '', ENDKEY => ''} 2023-04-24 09:00:19,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 09:00:19,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,409 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,410 DEBUG [StoreOpener-3695ff097ea6e04d645670eb37341990-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A 2023-04-24 09:00:19,410 DEBUG [StoreOpener-3695ff097ea6e04d645670eb37341990-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A 2023-04-24 09:00:19,411 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:19,411 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3695ff097ea6e04d645670eb37341990 columnFamilyName A 2023-04-24 09:00:19,411 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.HStore(310): Store=3695ff097ea6e04d645670eb37341990/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:19,411 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,413 DEBUG [StoreOpener-3695ff097ea6e04d645670eb37341990-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B 2023-04-24 09:00:19,413 DEBUG [StoreOpener-3695ff097ea6e04d645670eb37341990-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B 2023-04-24 09:00:19,413 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:19,413 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3695ff097ea6e04d645670eb37341990 columnFamilyName B 2023-04-24 09:00:19,413 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.HStore(310): Store=3695ff097ea6e04d645670eb37341990/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:19,414 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,414 DEBUG [StoreOpener-3695ff097ea6e04d645670eb37341990-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C 2023-04-24 09:00:19,415 DEBUG [StoreOpener-3695ff097ea6e04d645670eb37341990-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C 2023-04-24 09:00:19,415 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:19,415 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3695ff097ea6e04d645670eb37341990 columnFamilyName C 2023-04-24 09:00:19,415 INFO [StoreOpener-3695ff097ea6e04d645670eb37341990-1] regionserver.HStore(310): Store=3695ff097ea6e04d645670eb37341990/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:19,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 09:00:19,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 09:00:19,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 3695ff097ea6e04d645670eb37341990; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61649747, jitterRate=-0.08134718239307404}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 09:00:19,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:19,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., pid=49, masterSystemTime=1682326819405 2023-04-24 09:00:19,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,423 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=3695ff097ea6e04d645670eb37341990, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,423 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326819423"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326819423"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326819423"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326819423"}]},"ts":"1682326819423"} 2023-04-24 09:00:19,425 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-24 09:00:19,425 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 in 171 msec 2023-04-24 09:00:19,426 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-24 09:00:19,426 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3695ff097ea6e04d645670eb37341990, ASSIGN in 328 msec 2023-04-24 09:00:19,426 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 09:00:19,426 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326819426"}]},"ts":"1682326819426"} 2023-04-24 09:00:19,427 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 09:00:19,635 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 09:00:19,638 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.0170 sec 2023-04-24 09:00:19,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-24 09:00:19,729 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-24 09:00:19,730 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x4df55fe5 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,756 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ef6adaf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,759 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,763 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58420, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,764 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 09:00:19,766 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:51454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 09:00:19,768 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x0817e2e5 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,782 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f7a8e75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,782 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x4048dcb4 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,796 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@777f1e92, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,797 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x4e7a5e9a to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,815 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48d1969d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,815 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x62447900 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,830 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ed6755c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,830 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x519db5f2 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,840 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10505ab8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,840 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x609213af to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,855 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7205109f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,856 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x1ee0ad26 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,873 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e9fda3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,873 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x6acb8caa to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,888 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71d99548, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,888 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x70263db3 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,898 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46be51c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,899 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x25d2db45 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:19,913 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64d8b1e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:19,922 DEBUG [hconnection-0xe9b7743-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,922 DEBUG [hconnection-0x76d03e3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,922 DEBUG [hconnection-0x6171a94d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,923 DEBUG [hconnection-0x26e28ef3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:19,923 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,923 DEBUG [hconnection-0x117641be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,925 DEBUG [hconnection-0x4150e0b6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,925 DEBUG [hconnection-0x7f0d8c29-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,926 DEBUG [hconnection-0x72678ea9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,926 DEBUG [hconnection-0x6a8f25-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,926 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,927 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,931 DEBUG [hconnection-0x178d2f9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:19,933 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,933 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57178, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,934 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,934 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57190, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,934 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,934 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57224, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:19,938 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:19,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:19,941 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:19,941 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:19,941 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:19,941 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:19,943 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:19,944 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:19,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:19,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 09:00:19,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:19,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:19,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:19,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:19,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:19,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:19,953 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,953 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:19,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:19,954 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,954 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:19,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:19,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:19,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:19,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:19,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:19,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:19,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:19,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:19,959 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:19,959 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,959 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:19,959 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:19,960 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:19,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:19,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:19,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:19,960 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:19,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/4ea9df232bc34eee903558f7809a4aa3 2023-04-24 09:00:19,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326879967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326879968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:19,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326879968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:19,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326879969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:19,970 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:19,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326879970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:19,970 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:19,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:19,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d4bb180[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:19,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:19,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:19,980 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:19,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:19,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:19,981 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,981 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:19,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:19,981 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:19,989 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:19,989 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:19,989 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:19,989 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:19,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:19,991 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:19,991 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:19,991 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:19,991 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:19,991 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:19,997 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:19,997 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:19,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:19,997 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:19,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:19,999 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,999 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,005 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,005 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:20,005 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:20,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:20,005 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:20,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:20,005 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:20,006 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:20,006 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,006 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:20,006 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:20,006 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,006 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326880069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326880069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326880069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326880069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326880071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,106 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:20,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:20,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:20,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:20,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:20,112 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:20,112 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:20,112 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:20,112 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:20,112 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,112 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,122 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,122 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:20,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:20,122 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,122 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:20,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:20,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:20,123 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:20,123 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:20,123 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:20,123 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:20,123 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:20,123 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:20,123 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:20,123 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:20,124 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:20,124 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:20,124 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:20,124 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,124 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:20,124 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:20,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,130 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:20,131 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,131 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:20,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:20,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d08322c[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:20,132 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:20,132 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,139 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,139 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:20,139 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:20,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:20,139 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:20,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:20,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:20,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:20,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:20,149 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:20,149 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:20,149 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:20,149 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:20,149 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:20,155 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,155 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,155 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:20,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,156 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:20,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,156 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:20,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:20,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:20,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:20,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:20,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,164 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,164 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,164 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,264 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:20,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:20,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:20,267 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:20,267 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:20,267 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:20,267 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:20,267 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,267 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326880272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326880273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326880274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326880274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326880276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,305 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:20,305 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:20,305 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,306 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:20,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:20,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:20,307 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:20,307 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:20,307 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:20,307 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:20,307 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:20,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,322 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:20,322 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,322 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:20,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:20,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:20,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fdb8d31[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:20,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,330 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,331 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:20,331 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:20,331 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,331 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:20,331 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:20,339 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,339 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:20,339 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:20,339 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:20,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:20,340 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:20,340 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:20,340 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:20,341 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:20,341 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:20,347 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,347 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:20,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,347 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,347 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:20,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,348 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:20,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:20,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:20,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:20,355 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:20,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,356 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:20,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:20,356 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,356 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:20,356 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:20,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:20,356 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,356 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:20,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/5ebf45994f3044eb8c91eb2968522be4 2023-04-24 09:00:20,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/715ae55b03b047fda5816e8f075b05b6 2023-04-24 09:00:20,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/4ea9df232bc34eee903558f7809a4aa3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4ea9df232bc34eee903558f7809a4aa3 2023-04-24 09:00:20,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4ea9df232bc34eee903558f7809a4aa3, entries=250, sequenceid=14, filesize=16.2 K 2023-04-24 09:00:20,390 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/5ebf45994f3044eb8c91eb2968522be4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/5ebf45994f3044eb8c91eb2968522be4 2023-04-24 09:00:20,394 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/5ebf45994f3044eb8c91eb2968522be4, entries=150, sequenceid=14, filesize=11.6 K 2023-04-24 09:00:20,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/715ae55b03b047fda5816e8f075b05b6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/715ae55b03b047fda5816e8f075b05b6 2023-04-24 09:00:20,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/715ae55b03b047fda5816e8f075b05b6, entries=150, sequenceid=14, filesize=11.6 K 2023-04-24 09:00:20,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 3695ff097ea6e04d645670eb37341990 in 448ms, sequenceid=14, compaction requested=false 2023-04-24 09:00:20,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:20,456 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:20,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:20,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:20,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:20,461 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:20,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:20,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:20,461 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:20,462 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,462 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:20,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:20,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:20,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:20,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:20,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:20,473 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:20,473 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:20,473 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:20,474 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:20,474 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:20,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:20,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:20,474 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 09:00:20,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:20,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:20,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:20,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:20,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:20,474 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:20,481 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/6b67cc97903547339a6bac19c0e2c94d 2023-04-24 09:00:20,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:20,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:20,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326880585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326880586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326880587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326880587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326880588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326880690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326880690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326880692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326880692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326880693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326880893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326880894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326880895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326880895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:20,898 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/42dcfe49f706446599f1372caa90d712 2023-04-24 09:00:20,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:20,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326880897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326881195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326881199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326881200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326881200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326881202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,306 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/7097c984a8664405bfcf3b0a9c498669 2023-04-24 09:00:21,309 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/6b67cc97903547339a6bac19c0e2c94d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/6b67cc97903547339a6bac19c0e2c94d 2023-04-24 09:00:21,315 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/6b67cc97903547339a6bac19c0e2c94d, entries=150, sequenceid=34, filesize=11.6 K 2023-04-24 09:00:21,316 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/42dcfe49f706446599f1372caa90d712 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/42dcfe49f706446599f1372caa90d712 2023-04-24 09:00:21,331 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/42dcfe49f706446599f1372caa90d712, entries=150, sequenceid=34, filesize=11.6 K 2023-04-24 09:00:21,332 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/7097c984a8664405bfcf3b0a9c498669 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/7097c984a8664405bfcf3b0a9c498669 2023-04-24 09:00:21,337 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/7097c984a8664405bfcf3b0a9c498669, entries=150, sequenceid=34, filesize=11.6 K 2023-04-24 09:00:21,338 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3695ff097ea6e04d645670eb37341990 in 864ms, sequenceid=34, compaction requested=false 2023-04-24 09:00:21,338 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:21,338 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:21,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:21,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:21,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,339 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:21,339 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:21,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:21,372 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:21,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:21,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:21,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:21,374 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:21,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5451bf9c[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:21,374 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:21,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:21,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:21,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:21,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:21,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:21,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:21,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:21,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:21,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:21,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:21,390 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:21,391 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:21,391 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:21,391 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:21,391 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:21,397 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,397 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,397 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:21,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:21,397 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:21,398 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:21,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:21,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:21,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:21,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:21,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:21,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:21,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:21,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:21,406 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:21,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:21,406 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:21,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:21,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:21,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:21,406 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,506 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:21,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:21,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:21,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:21,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:21,508 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:21,508 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:21,509 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:21,509 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:21,509 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,509 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:21,514 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:21,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:21,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:21,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:21,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:21,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:21,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:21,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:21,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:21,515 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:21,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:21,516 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:21,516 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:21,516 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:21,516 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 09:00:21,516 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:21,516 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:21,516 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:21,516 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:21,516 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:21,517 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:21,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:21,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:21,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326881721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326881724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326881727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326881728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326881728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326881829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326881833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326881833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326881837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:21,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326881837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:21,930 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/eb4e0f4a1c234db397e623a904debcc9 2023-04-24 09:00:22,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326882034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326882036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326882037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326882041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326882043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,339 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/285f2fb94edd4a989499191c2901acaf 2023-04-24 09:00:22,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326882339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326882341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326882342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326882346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326882347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,747 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/94f03489dfe740ad92ba7974aab22cce 2023-04-24 09:00:22,750 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/eb4e0f4a1c234db397e623a904debcc9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/eb4e0f4a1c234db397e623a904debcc9 2023-04-24 09:00:22,753 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/eb4e0f4a1c234db397e623a904debcc9, entries=150, sequenceid=46, filesize=11.6 K 2023-04-24 09:00:22,753 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/285f2fb94edd4a989499191c2901acaf as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/285f2fb94edd4a989499191c2901acaf 2023-04-24 09:00:22,756 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/285f2fb94edd4a989499191c2901acaf, entries=150, sequenceid=46, filesize=11.6 K 2023-04-24 09:00:22,757 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/94f03489dfe740ad92ba7974aab22cce as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/94f03489dfe740ad92ba7974aab22cce 2023-04-24 09:00:22,759 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/94f03489dfe740ad92ba7974aab22cce, entries=150, sequenceid=46, filesize=11.6 K 2023-04-24 09:00:22,760 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3695ff097ea6e04d645670eb37341990 in 1244ms, sequenceid=46, compaction requested=true 2023-04-24 09:00:22,760 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:22,760 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:22,760 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:22,760 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:22,760 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,760 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:22,760 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:22,770 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,770 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:22,770 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,770 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:22,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:22,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6426c327[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:22,771 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:22,771 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:22,781 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:22,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:22,781 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:22,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:22,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:22,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:22,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:22,791 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:22,791 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:22,791 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:22,791 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:22,791 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:22,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:22,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:22,798 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:22,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:22,805 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:22,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:22,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:22,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:22,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:22,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:22,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:22,806 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:22,806 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:22,806 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:22,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:22,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-24 09:00:22,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:22,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:22,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:22,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:22,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:22,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:22,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326882861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326882861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326882862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326882863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326882865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,906 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:22,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:22,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:22,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:22,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:22,909 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:22,909 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:22,909 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:22,909 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:22,909 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,909 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:22,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:22,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:22,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:22,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:22,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:22,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:22,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:22,915 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:22,915 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:22,915 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:22,915 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:22,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:22,916 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,916 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:22,916 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:22,942 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,942 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:22,942 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,942 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:22,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:22,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e314dd[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:22,943 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:22,943 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:22,955 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:22,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:22,956 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:22,964 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:22,964 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:22,964 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,964 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:22,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:22,966 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:22,966 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:22,966 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:22,966 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:22,966 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:22,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:22,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:22,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:22,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:22,973 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,974 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326882971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326882971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326882971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326882974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:22,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326882975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:22,987 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:22,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:22,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:22,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:22,987 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:22,987 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:22,987 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:22,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,087 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:23,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:23,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:23,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:23,090 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:23,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:23,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:23,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,103 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,104 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:23,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:23,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:23,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:23,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:23,104 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:23,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:23,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:23,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:23,105 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:23,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:23,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:23,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:23,114 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,114 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,114 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,114 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:23,115 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:23,115 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf2b4f5[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:23,122 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,122 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,122 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:23,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:23,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:23,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:23,130 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:23,130 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:23,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,130 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:23,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:23,132 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:23,132 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:23,132 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:23,132 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:23,132 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:23,139 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,139 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,139 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,141 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,147 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:23,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:23,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:23,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:23,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:23,147 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:23,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,147 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:23,147 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,148 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326883177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326883177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326883177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326883180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326883181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,248 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:23,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:23,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:23,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:23,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:23,250 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:23,250 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,250 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:23,250 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:23,251 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,251 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,255 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,255 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,256 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:23,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:23,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:23,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:23,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:23,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:23,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:23,257 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,257 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:23,257 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:23,257 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:23,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:23,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:23,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:23,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/454a1277af684f949d968a1f8ea301e5 2023-04-24 09:00:23,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:23,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42864548[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:23,265 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:23,266 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/56b7f49bbee84b16b826dccb69456a1d 2023-04-24 09:00:23,273 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,273 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:23,273 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:23,273 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:23,273 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,273 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:23,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:23,281 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:23,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:23,282 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:23,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:23,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:23,283 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:23,283 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:23,289 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,289 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:23,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,290 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,291 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,293 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:23,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:23,298 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:23,298 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:23,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:23,398 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:23,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:23,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:23,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:23,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:23,401 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:23,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:23,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:23,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,401 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,406 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,406 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:23,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:23,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:23,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:23,407 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,407 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:23,407 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:23,407 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:23,407 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:23,414 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:23,415 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:23,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106b99c4[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:23,415 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:23,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:23,422 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,422 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:23,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:23,451 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:23,451 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,452 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:23,452 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:23,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:23,453 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:23,454 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:23,454 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:23,454 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:23,454 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:23,464 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,464 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,464 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:23,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,464 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,465 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:23,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:23,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:23,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:23,472 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:23,472 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:23,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,473 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:23,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,473 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326883481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326883481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326883481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326883482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:23,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326883489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,573 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:23,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:23,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:23,576 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:23,576 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,576 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:23,576 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:23,576 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,576 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,587 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,587 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,587 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,587 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,587 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 09:00:23,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:23,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:23,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:23,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:23,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,592 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:23,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:23,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:23,593 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,593 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:23,593 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:23,593 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:23,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:23,593 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,594 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:23,594 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:23,603 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,603 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,604 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,604 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:23,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@308e22e8[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:23,605 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:23,605 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,614 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:23,614 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:23,614 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:23,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:23,622 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:23,622 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:23,622 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:23,622 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:23,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:23,624 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:23,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:23,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:23,624 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:23,624 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:23,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:23,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:23,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:23,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:23,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:23,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:23,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:23,639 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:23,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,639 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:23,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,686 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/e2ed58a69f884ae986312c7fc1009e64 2023-04-24 09:00:23,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/454a1277af684f949d968a1f8ea301e5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/454a1277af684f949d968a1f8ea301e5 2023-04-24 09:00:23,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/454a1277af684f949d968a1f8ea301e5, entries=250, sequenceid=68, filesize=16.2 K 2023-04-24 09:00:23,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/56b7f49bbee84b16b826dccb69456a1d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/56b7f49bbee84b16b826dccb69456a1d 2023-04-24 09:00:23,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/56b7f49bbee84b16b826dccb69456a1d, entries=150, sequenceid=68, filesize=11.6 K 2023-04-24 09:00:23,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/e2ed58a69f884ae986312c7fc1009e64 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e2ed58a69f884ae986312c7fc1009e64 2023-04-24 09:00:23,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e2ed58a69f884ae986312c7fc1009e64, entries=150, sequenceid=68, filesize=11.6 K 2023-04-24 09:00:23,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=46.96 KB/48090 for 3695ff097ea6e04d645670eb37341990 in 847ms, sequenceid=68, compaction requested=true 2023-04-24 09:00:23,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:23,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:23,698 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:23,698 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:23,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:23,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:23,700 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56904 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:23,700 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/A is initiating minor compaction (all files) 2023-04-24 09:00:23,700 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/A in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,700 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4ea9df232bc34eee903558f7809a4aa3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/6b67cc97903547339a6bac19c0e2c94d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/eb4e0f4a1c234db397e623a904debcc9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/454a1277af684f949d968a1f8ea301e5] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=55.6 K 2023-04-24 09:00:23,700 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:23,700 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/B is initiating minor compaction (all files) 2023-04-24 09:00:23,700 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/B in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,700 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/5ebf45994f3044eb8c91eb2968522be4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/42dcfe49f706446599f1372caa90d712, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/285f2fb94edd4a989499191c2901acaf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/56b7f49bbee84b16b826dccb69456a1d] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=46.4 K 2023-04-24 09:00:23,701 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 4ea9df232bc34eee903558f7809a4aa3, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326819948 2023-04-24 09:00:23,701 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5ebf45994f3044eb8c91eb2968522be4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326819948 2023-04-24 09:00:23,701 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 6b67cc97903547339a6bac19c0e2c94d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682326819965 2023-04-24 09:00:23,701 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 42dcfe49f706446599f1372caa90d712, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682326819965 2023-04-24 09:00:23,701 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting eb4e0f4a1c234db397e623a904debcc9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682326820581 2023-04-24 09:00:23,701 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 285f2fb94edd4a989499191c2901acaf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682326820581 2023-04-24 09:00:23,701 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 454a1277af684f949d968a1f8ea301e5, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682326821716 2023-04-24 09:00:23,701 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 56b7f49bbee84b16b826dccb69456a1d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682326821716 2023-04-24 09:00:23,707 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#B#compaction#348 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:23,709 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#A#compaction#349 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:23,739 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:23,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:23,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:23,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:23,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:23,742 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:23,742 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,743 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:23,743 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:23,743 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,743 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:23,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:23,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:23,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:23,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:23,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:23,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:23,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:23,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:23,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:23,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:23,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:23,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:23,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:23,749 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 09:00:23,749 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:23,749 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:23,749 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:23,749 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:23,749 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:23,749 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:23,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:23,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:24,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326823998, max timestamp=1682326824010Num uniques 859; 2023-04-24 09:00:24,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326823998, max timestamp=1682326824010Num uniques 859; 2023-04-24 09:00:24,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:24,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:24,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326823998, max timestamp=1682326824010Num uniques 859; 2023-04-24 09:00:24,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:24,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326884027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326884028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326884029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326884029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326884031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,115 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/b7bd1cdcaea14439a1676e5f0fb05715 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/b7bd1cdcaea14439a1676e5f0fb05715 2023-04-24 09:00:24,116 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/368291fdc5c84be18d214db6c12135ca as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/368291fdc5c84be18d214db6c12135ca 2023-04-24 09:00:24,118 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3695ff097ea6e04d645670eb37341990/B of 3695ff097ea6e04d645670eb37341990 into b7bd1cdcaea14439a1676e5f0fb05715(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:24,118 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:24,118 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/B, priority=12, startTime=1682326823698; duration=0sec 2023-04-24 09:00:24,118 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:24,118 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:24,119 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3695ff097ea6e04d645670eb37341990/A of 3695ff097ea6e04d645670eb37341990 into 368291fdc5c84be18d214db6c12135ca(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:24,119 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:24,119 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/A, priority=12, startTime=1682326823698; duration=0sec 2023-04-24 09:00:24,119 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:24,119 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:24,119 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/C is initiating minor compaction (all files) 2023-04-24 09:00:24,119 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/C in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:24,119 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/715ae55b03b047fda5816e8f075b05b6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/7097c984a8664405bfcf3b0a9c498669, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/94f03489dfe740ad92ba7974aab22cce, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e2ed58a69f884ae986312c7fc1009e64] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=46.4 K 2023-04-24 09:00:24,120 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 715ae55b03b047fda5816e8f075b05b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682326819948 2023-04-24 09:00:24,120 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7097c984a8664405bfcf3b0a9c498669, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682326819965 2023-04-24 09:00:24,120 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 94f03489dfe740ad92ba7974aab22cce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682326820581 2023-04-24 09:00:24,123 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e2ed58a69f884ae986312c7fc1009e64, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682326821716 2023-04-24 09:00:24,129 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#C#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326884134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326884136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326884136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326884137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326884138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,159 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/814e0716e1fa494a99b3710f47c60c14 2023-04-24 09:00:24,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326884339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326884339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326884339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326884339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326884344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,535 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/398cdc841a974933a2525f23da2cb610 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/398cdc841a974933a2525f23da2cb610 2023-04-24 09:00:24,538 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3695ff097ea6e04d645670eb37341990/C of 3695ff097ea6e04d645670eb37341990 into 398cdc841a974933a2525f23da2cb610(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:24,538 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:24,538 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/C, priority=12, startTime=1682326823698; duration=0sec 2023-04-24 09:00:24,538 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:24,567 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/f737a999301c4bb2b552763f489c3bd3 2023-04-24 09:00:24,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326884644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326884644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326884645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326884646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:24,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326884650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,974 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/9e41673e73c7423ab5d1e4428985467b 2023-04-24 09:00:24,977 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/814e0716e1fa494a99b3710f47c60c14 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/814e0716e1fa494a99b3710f47c60c14 2023-04-24 09:00:24,980 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/814e0716e1fa494a99b3710f47c60c14, entries=100, sequenceid=78, filesize=9.3 K 2023-04-24 09:00:24,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/f737a999301c4bb2b552763f489c3bd3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f737a999301c4bb2b552763f489c3bd3 2023-04-24 09:00:24,983 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f737a999301c4bb2b552763f489c3bd3, entries=100, sequenceid=78, filesize=9.3 K 2023-04-24 09:00:24,983 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/9e41673e73c7423ab5d1e4428985467b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/9e41673e73c7423ab5d1e4428985467b 2023-04-24 09:00:24,986 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/9e41673e73c7423ab5d1e4428985467b, entries=100, sequenceid=78, filesize=9.3 K 2023-04-24 09:00:24,987 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3695ff097ea6e04d645670eb37341990 in 1238ms, sequenceid=78, compaction requested=false 2023-04-24 09:00:24,987 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:24,987 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:24,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:24,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:24,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:24,987 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:24,997 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,997 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:24,997 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:24,997 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:24,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:24,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e557d10[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:24,998 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:24,998 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:25,005 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:25,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:25,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:25,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:25,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,005 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:25,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:25,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:25,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:25,014 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,014 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:25,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:25,015 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:25,015 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:25,015 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:25,016 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:25,016 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:25,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:25,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:25,022 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:25,023 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:25,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:25,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:25,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:25,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:25,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:25,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:25,031 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:25,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:25,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:25,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:25,031 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:25,031 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:25,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:25,131 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:25,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:25,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:25,133 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:25,133 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:25,134 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:25,134 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:25,134 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,134 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:25,138 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:25,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:25,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,139 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:25,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:25,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:25,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:25,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:25,140 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:25,140 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:25,140 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:25,140 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:25,140 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:25,140 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:25,140 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:25,140 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-24 09:00:25,140 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:25,140 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:25,140 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:25,140 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:25,140 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:25,140 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:25,154 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:25,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326885156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326885156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326885161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326885161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326885162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326885263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326885264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326885268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326885268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326885468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326885468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326885472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326885473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,552 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/e86d1345e5f6482fa7e3016b6ccb7633 2023-04-24 09:00:25,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326885772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326885773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326885779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:25,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326885780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:25,961 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/7a8911391d5e45a2923d95b89a1766ac 2023-04-24 09:00:26,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:26,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326886172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:26,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326886274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:26,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326886293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:26,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326886293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:26,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326886293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,370 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/3e90baa41d2540c3980e542c1e8351bd 2023-04-24 09:00:26,373 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/e86d1345e5f6482fa7e3016b6ccb7633 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e86d1345e5f6482fa7e3016b6ccb7633 2023-04-24 09:00:26,377 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e86d1345e5f6482fa7e3016b6ccb7633, entries=150, sequenceid=116, filesize=11.6 K 2023-04-24 09:00:26,378 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/7a8911391d5e45a2923d95b89a1766ac as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7a8911391d5e45a2923d95b89a1766ac 2023-04-24 09:00:26,381 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7a8911391d5e45a2923d95b89a1766ac, entries=150, sequenceid=116, filesize=11.6 K 2023-04-24 09:00:26,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/3e90baa41d2540c3980e542c1e8351bd as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/3e90baa41d2540c3980e542c1e8351bd 2023-04-24 09:00:26,384 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/3e90baa41d2540c3980e542c1e8351bd, entries=150, sequenceid=116, filesize=11.6 K 2023-04-24 09:00:26,384 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 3695ff097ea6e04d645670eb37341990 in 1244ms, sequenceid=116, compaction requested=true 2023-04-24 09:00:26,384 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:26,384 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:26,395 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:26,395 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:26,395 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:26,395 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:26,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:26,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e09d397[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:26,396 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:26,396 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:26,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:26,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:26,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:26,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:26,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:26,406 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:26,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,413 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:26,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:26,414 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:26,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:26,415 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:26,416 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:26,416 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:26,416 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:26,416 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:26,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:26,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:26,441 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:26,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:26,465 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:26,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:26,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:26,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:26,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:26,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:26,465 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:26,465 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:26,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:26,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:26,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:26,465 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:26,466 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,565 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:26,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:26,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:26,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:26,568 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:26,568 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:26,568 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:26,568 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:26,569 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,569 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:26,580 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:26,580 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:26,581 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:26,581 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:26,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:26,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:26,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:26,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:26,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:26,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:26,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:26,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:26,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:26,582 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:26,582 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:26,582 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 09:00:26,582 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:26,582 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:26,582 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:26,582 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:26,582 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:26,582 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:26,988 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/c30b4ce9e5e84d6ab967d458db11dd2e 2023-04-24 09:00:27,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:27,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:27,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682326827302, max timestamp=1682326827323Num uniques 859; 2023-04-24 09:00:27,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-24 09:00:27,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682326827302, max timestamp=1682326827323Num uniques 859; 2023-04-24 09:00:27,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-24 09:00:27,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682326827302, max timestamp=1682326827323Num uniques 859; 2023-04-24 09:00:27,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-24 09:00:27,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326887341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326887342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326887342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326887343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,395 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/ee2b74f0edb34f98bb371f8884d2dabd 2023-04-24 09:00:27,403 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/718dfa3e18144d75b878861bc26a84e1 2023-04-24 09:00:27,406 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/c30b4ce9e5e84d6ab967d458db11dd2e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/c30b4ce9e5e84d6ab967d458db11dd2e 2023-04-24 09:00:27,408 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/c30b4ce9e5e84d6ab967d458db11dd2e, entries=100, sequenceid=126, filesize=9.3 K 2023-04-24 09:00:27,409 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/ee2b74f0edb34f98bb371f8884d2dabd as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ee2b74f0edb34f98bb371f8884d2dabd 2023-04-24 09:00:27,412 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ee2b74f0edb34f98bb371f8884d2dabd, entries=100, sequenceid=126, filesize=9.3 K 2023-04-24 09:00:27,412 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/718dfa3e18144d75b878861bc26a84e1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/718dfa3e18144d75b878861bc26a84e1 2023-04-24 09:00:27,415 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/718dfa3e18144d75b878861bc26a84e1, entries=100, sequenceid=126, filesize=9.3 K 2023-04-24 09:00:27,415 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 3695ff097ea6e04d645670eb37341990 in 833ms, sequenceid=126, compaction requested=true 2023-04-24 09:00:27,415 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:27,415 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:27,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:27,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:27,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:27,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:27,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:27,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=147.60 KB heapSize=401.39 KB 2023-04-24 09:00:27,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:27,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:27,453 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,453 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:27,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,453 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:27,454 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,454 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:27,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e0d62b[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:27,455 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:27,455 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,464 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:27,464 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:27,464 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:27,464 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,464 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:27,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326887457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326887464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:27,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:27,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:27,474 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:27,474 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:27,474 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:27,474 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:27,474 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:27,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326887464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326887465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:27,481 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,487 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:27,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,497 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:27,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:27,497 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:27,497 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:27,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326887568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326887569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326887576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326887576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,597 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:27,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:27,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:27,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:27,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:27,600 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:27,600 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:27,600 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:27,601 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:27,601 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,601 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,605 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:27,605 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:27,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:27,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:27,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:27,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:27,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:27,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:27,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:27,607 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:27,607 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:27,607 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:27,607 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:27,608 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:27,608 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:27,608 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:27,608 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,608 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:27,608 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:27,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,613 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,614 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,614 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:27,615 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:27,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43b83dfd[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:27,615 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,622 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,622 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,622 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:27,622 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:27,622 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:27,622 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:27,630 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:27,630 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:27,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,630 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:27,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:27,632 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:27,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:27,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:27,632 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:27,633 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:27,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:27,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,640 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:27,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:27,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:27,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,647 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:27,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,647 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,747 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:27,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:27,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:27,751 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:27,751 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:27,751 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:27,751 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:27,751 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,751 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,755 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:27,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:27,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:27,755 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,755 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:27,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:27,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:27,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:27,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:27,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:27,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:27,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:27,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:27,756 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:27,756 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:27,757 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:27,757 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:27,757 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:27,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,764 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,764 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:27,765 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:27,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56907567[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:27,765 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,772 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,772 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:27,772 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,772 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:27,772 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:27,772 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:27,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:27,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:27,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:27,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326887774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326887775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:27,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:27,782 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:27,782 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:27,782 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:27,782 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:27,782 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:27,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326887781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,789 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:27,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:27,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326887782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:27,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:27,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:27,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:27,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,797 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:27,797 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:27,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:27,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/5bedd0acda4f4d8a9fdd95e8216f21f8 2023-04-24 09:00:27,898 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:27,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:27,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:27,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:27,900 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:27,900 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:27,900 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:27,900 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:27,900 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,901 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:27,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:27,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:27,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:27,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:27,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:27,906 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:27,907 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:27,907 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:27,907 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:27,907 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:27,907 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:27,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:27,915 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:27,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56859149[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:27,915 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,922 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:27,922 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,922 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:27,922 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:27,922 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:27,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:27,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:27,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:27,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:27,932 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:27,932 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:27,932 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:27,932 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:27,932 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:27,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,939 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:27,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:27,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:27,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:27,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:27,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:27,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:27,947 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:27,947 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:28,048 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:28,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:28,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:28,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:28,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:28,051 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:28,051 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:28,051 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:28,051 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:28,051 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,051 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,055 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:28,055 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,055 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:28,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:28,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:28,056 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:28,056 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:28,056 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:28,057 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:28,057 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:28,057 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:28,057 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:28,057 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:28,063 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:28,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:28,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:28,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:28,065 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:28,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5250443c[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:28,065 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:28,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:28,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:28,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:28,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:28,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:28,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:28,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:28,082 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:28,083 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:28,083 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:28,083 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:28,083 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:28,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326888081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:28,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326888082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:28,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:28,089 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:28,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326888089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326888091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:28,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:28,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:28,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:28,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:28,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,097 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:28,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:28,097 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:28,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:28,097 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:28,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,098 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,098 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326888181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,186 DEBUG [Thread-2662] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:28,197 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:28,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:28,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:28,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:28,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:28,200 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:28,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:28,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:28,200 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:28,201 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,201 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,205 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:28,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:28,205 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,205 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:28,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:28,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:28,206 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:28,206 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:28,206 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:28,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:28,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:28,207 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:28,207 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:28,207 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:28,207 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:28,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:28,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:28,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:28,207 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:28,213 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,213 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:28,214 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,214 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:28,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:28,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:28,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5085dc5e[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:28,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,222 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:28,222 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,222 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:28,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:28,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:28,230 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:28,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,230 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:28,230 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:28,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:28,232 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:28,232 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:28,232 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:28,232 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:28,232 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:28,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:28,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:28,239 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,240 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:28,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:28,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:28,266 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,266 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,266 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:28,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:28,266 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/cf3a79678eb74f88b3ff72b44c893655 2023-04-24 09:00:28,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/dbad1b82bf684c59a045a97d87653092 2023-04-24 09:00:28,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/5bedd0acda4f4d8a9fdd95e8216f21f8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/5bedd0acda4f4d8a9fdd95e8216f21f8 2023-04-24 09:00:28,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/5bedd0acda4f4d8a9fdd95e8216f21f8, entries=150, sequenceid=161, filesize=11.8 K 2023-04-24 09:00:28,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/cf3a79678eb74f88b3ff72b44c893655 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/cf3a79678eb74f88b3ff72b44c893655 2023-04-24 09:00:28,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/cf3a79678eb74f88b3ff72b44c893655, entries=150, sequenceid=161, filesize=11.8 K 2023-04-24 09:00:28,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/dbad1b82bf684c59a045a97d87653092 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dbad1b82bf684c59a045a97d87653092 2023-04-24 09:00:28,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dbad1b82bf684c59a045a97d87653092, entries=150, sequenceid=161, filesize=11.8 K 2023-04-24 09:00:28,305 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~421.27 KB/431376, currentSize=33.54 KB/34350 for 3695ff097ea6e04d645670eb37341990 in 853ms, sequenceid=161, compaction requested=true 2023-04-24 09:00:28,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:28,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:28,305 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 09:00:28,305 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 09:00:28,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:28,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:28,308 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55030 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 09:00:28,308 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55030 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 09:00:28,308 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/A is initiating minor compaction (all files) 2023-04-24 09:00:28,308 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/B is initiating minor compaction (all files) 2023-04-24 09:00:28,308 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/A in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:28,308 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/B in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:28,308 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/368291fdc5c84be18d214db6c12135ca, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/814e0716e1fa494a99b3710f47c60c14, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e86d1345e5f6482fa7e3016b6ccb7633, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/c30b4ce9e5e84d6ab967d458db11dd2e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/5bedd0acda4f4d8a9fdd95e8216f21f8] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=53.7 K 2023-04-24 09:00:28,308 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/b7bd1cdcaea14439a1676e5f0fb05715, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f737a999301c4bb2b552763f489c3bd3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7a8911391d5e45a2923d95b89a1766ac, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ee2b74f0edb34f98bb371f8884d2dabd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/cf3a79678eb74f88b3ff72b44c893655] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=53.7 K 2023-04-24 09:00:28,318 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting b7bd1cdcaea14439a1676e5f0fb05715, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682326821716 2023-04-24 09:00:28,318 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 368291fdc5c84be18d214db6c12135ca, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682326821716 2023-04-24 09:00:28,319 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f737a999301c4bb2b552763f489c3bd3, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1682326822860 2023-04-24 09:00:28,320 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 814e0716e1fa494a99b3710f47c60c14, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1682326822860 2023-04-24 09:00:28,320 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7a8911391d5e45a2923d95b89a1766ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682326824026 2023-04-24 09:00:28,320 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e86d1345e5f6482fa7e3016b6ccb7633, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682326824026 2023-04-24 09:00:28,320 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting ee2b74f0edb34f98bb371f8884d2dabd, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682326825160 2023-04-24 09:00:28,320 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting c30b4ce9e5e84d6ab967d458db11dd2e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682326825160 2023-04-24 09:00:28,321 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting cf3a79678eb74f88b3ff72b44c893655, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682326827339 2023-04-24 09:00:28,321 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 5bedd0acda4f4d8a9fdd95e8216f21f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682326827339 2023-04-24 09:00:28,337 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#B#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:28,345 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#A#compaction#364 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:28,366 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:28,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:28,371 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/2ecda413122b4489a6e17f518b7feda5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/2ecda413122b4489a6e17f518b7feda5 2023-04-24 09:00:28,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:28,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:28,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:28,372 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:28,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:28,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:28,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:28,373 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,373 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,374 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/e27287df98ad4c2b923f93854106f6d1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e27287df98ad4c2b923f93854106f6d1 2023-04-24 09:00:28,378 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3695ff097ea6e04d645670eb37341990/B of 3695ff097ea6e04d645670eb37341990 into 2ecda413122b4489a6e17f518b7feda5(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:28,378 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:28,378 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/B, priority=11, startTime=1682326828305; duration=0sec 2023-04-24 09:00:28,379 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:28,379 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-24 09:00:28,385 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55030 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-24 09:00:28,385 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/C is initiating minor compaction (all files) 2023-04-24 09:00:28,385 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/C in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:28,385 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/398cdc841a974933a2525f23da2cb610, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/9e41673e73c7423ab5d1e4428985467b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/3e90baa41d2540c3980e542c1e8351bd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/718dfa3e18144d75b878861bc26a84e1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dbad1b82bf684c59a045a97d87653092] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=53.7 K 2023-04-24 09:00:28,386 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 398cdc841a974933a2525f23da2cb610, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682326821716 2023-04-24 09:00:28,386 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3695ff097ea6e04d645670eb37341990/A of 3695ff097ea6e04d645670eb37341990 into e27287df98ad4c2b923f93854106f6d1(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:28,386 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:28,386 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/A, priority=11, startTime=1682326828305; duration=0sec 2023-04-24 09:00:28,386 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:28,387 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,387 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:28,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:28,387 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,388 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:28,388 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 9e41673e73c7423ab5d1e4428985467b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1682326822860 2023-04-24 09:00:28,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:28,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:28,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:28,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:28,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:28,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:28,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:28,389 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3e90baa41d2540c3980e542c1e8351bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1682326824026 2023-04-24 09:00:28,389 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 718dfa3e18144d75b878861bc26a84e1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682326825160 2023-04-24 09:00:28,390 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting dbad1b82bf684c59a045a97d87653092, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682326827339 2023-04-24 09:00:28,400 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:28,400 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:28,402 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#C#compaction#365 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:28,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:28,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:28,404 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 09:00:28,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:28,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:28,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:28,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:28,404 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:28,405 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:28,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:28,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:28,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326828609, max timestamp=1682326828619Num uniques 859; 2023-04-24 09:00:28,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:28,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326828609, max timestamp=1682326828619Num uniques 859; 2023-04-24 09:00:28,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:28,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326828609, max timestamp=1682326828619Num uniques 859; 2023-04-24 09:00:28,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:28,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682326828632, max timestamp=1682326828653Num uniques 1068; 2023-04-24 09:00:28,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682326828632, max timestamp=1682326828653Num uniques 1068; 2023-04-24 09:00:28,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1682326828632, max timestamp=1682326828653Num uniques 1068; 2023-04-24 09:00:28,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-24 09:00:28,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-24 09:00:28,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-24 09:00:28,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326888681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326888685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326888685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326888686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326888788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326888788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326888794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326888798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,816 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/e1b5b4f241604d1b89fcf3b9cf18b26e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e1b5b4f241604d1b89fcf3b9cf18b26e 2023-04-24 09:00:28,820 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3695ff097ea6e04d645670eb37341990/C of 3695ff097ea6e04d645670eb37341990 into e1b5b4f241604d1b89fcf3b9cf18b26e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:28,820 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:28,820 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/C, priority=11, startTime=1682326828305; duration=0sec 2023-04-24 09:00:28,820 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:28,826 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/9cbc0f2979b142f890b114f902ed79bb 2023-04-24 09:00:28,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326888994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:28,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:28,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326888994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326889000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326889005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,237 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/a769f546d0e34a5db56ae66702fccfb7 2023-04-24 09:00:29,243 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/676b2e94075140ed98baef23d6f53b8f 2023-04-24 09:00:29,246 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/9cbc0f2979b142f890b114f902ed79bb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9cbc0f2979b142f890b114f902ed79bb 2023-04-24 09:00:29,249 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9cbc0f2979b142f890b114f902ed79bb, entries=150, sequenceid=171, filesize=11.8 K 2023-04-24 09:00:29,249 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/a769f546d0e34a5db56ae66702fccfb7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a769f546d0e34a5db56ae66702fccfb7 2023-04-24 09:00:29,252 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a769f546d0e34a5db56ae66702fccfb7, entries=150, sequenceid=171, filesize=11.8 K 2023-04-24 09:00:29,253 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/676b2e94075140ed98baef23d6f53b8f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/676b2e94075140ed98baef23d6f53b8f 2023-04-24 09:00:29,255 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/676b2e94075140ed98baef23d6f53b8f, entries=150, sequenceid=171, filesize=11.8 K 2023-04-24 09:00:29,256 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 3695ff097ea6e04d645670eb37341990 in 852ms, sequenceid=171, compaction requested=false 2023-04-24 09:00:29,256 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:29,256 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:29,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:29,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:29,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:29,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:29,289 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,289 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,289 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,289 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:29,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eba064[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:29,290 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:29,290 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:29,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:29,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:29,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:29,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:29,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=161.02 KB heapSize=450.47 KB 2023-04-24 09:00:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:29,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:29,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:29,305 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/f576161bd0524272b57e3798de835dda 2023-04-24 09:00:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:29,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:29,308 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:29,309 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:29,309 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:29,309 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:29,309 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:29,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,314 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:29,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,315 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326889312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326889312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326889312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326889313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:29,322 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:29,322 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:29,322 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:29,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:29,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:29,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:29,422 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:29,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326889420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326889420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326889420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:29,425 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:29,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:29,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:29,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:29,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:29,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:29,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:29,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:29,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:29,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:29,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:29,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:29,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:29,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:29,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:29,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:29,432 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:29,432 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:29,432 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:29,432 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:29,432 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,432 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:29,432 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:29,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,438 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,439 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,439 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:29,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:29,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9e7649[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:29,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,447 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:29,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:29,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:29,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:29,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:29,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:29,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:29,457 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:29,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:29,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:29,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:29,457 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:29,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,464 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,465 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:29,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,480 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:29,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:29,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:29,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:29,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:29,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:29,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:29,481 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:29,481 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:29,481 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,481 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,481 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,581 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:29,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:29,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:29,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:29,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:29,584 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:29,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:29,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:29,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:29,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,588 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:29,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:29,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:29,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:29,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:29,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:29,590 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:29,590 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:29,590 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:29,590 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:29,590 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:29,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:29,598 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:29,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbfac22[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:29,598 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326889624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326889625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326889625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,715 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,716 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:29,716 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:29,716 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:29,716 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,716 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:29,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/8147be04f4bc4bde8750c829ebe1f691 2023-04-24 09:00:29,758 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:29,758 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:29,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:29,760 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:29,760 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:29,760 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:29,760 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:29,760 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:29,772 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,772 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,772 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:29,772 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:29,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:29,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:29,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,780 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:29,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:29,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:29,780 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:29,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:29,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:29,781 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:29,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,781 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326889819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,881 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:29,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:29,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:29,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:29,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:29,884 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:29,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:29,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:29,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:29,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,884 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,888 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:29,888 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:29,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:29,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:29,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:29,890 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:29,890 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:29,890 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:29,890 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:29,890 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:29,917 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,918 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,918 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:29,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a6e2a7[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:29,919 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:29,919 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:29,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:29,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:29,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:29,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57224 deadline: 1682326889929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57184 deadline: 1682326889930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:29,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57228 deadline: 1682326889930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:29,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,939 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:29,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:29,940 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:29,941 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:29,941 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:29,941 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:29,941 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:29,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:29,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:29,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:29,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:29,955 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:29,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:29,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:29,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:29,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:29,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:29,956 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,956 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:29,956 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:29,956 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:29,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:29,956 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:29,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:30,056 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:30,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:30,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:30,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:30,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:30,058 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:30,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:30,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:30,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:30,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,063 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:30,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:30,064 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,064 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:30,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:30,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:30,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:30,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:30,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:30,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:30,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:30,074 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:30,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9168077[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:30,074 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:30,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:30,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:30,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:30,088 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:30,088 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:30,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:30,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:30,090 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:30,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:30,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:30,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:30,090 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:30,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,097 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,105 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:30,105 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:30,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:30,105 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,105 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:30,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:30,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:30,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:30,105 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,106 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:30,106 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:30,106 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,106 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:30,106 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,106 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,106 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/dd87ef75633742089aa2844d24b9354a 2023-04-24 09:00:30,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/f576161bd0524272b57e3798de835dda as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f576161bd0524272b57e3798de835dda 2023-04-24 09:00:30,131 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f576161bd0524272b57e3798de835dda, entries=200, sequenceid=227, filesize=14.1 K 2023-04-24 09:00:30,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/8147be04f4bc4bde8750c829ebe1f691 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/8147be04f4bc4bde8750c829ebe1f691 2023-04-24 09:00:30,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/8147be04f4bc4bde8750c829ebe1f691, entries=150, sequenceid=227, filesize=11.8 K 2023-04-24 09:00:30,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/dd87ef75633742089aa2844d24b9354a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dd87ef75633742089aa2844d24b9354a 2023-04-24 09:00:30,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dd87ef75633742089aa2844d24b9354a, entries=150, sequenceid=227, filesize=11.8 K 2023-04-24 09:00:30,156 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~450.42 KB/461232, currentSize=26.84 KB/27480 for 3695ff097ea6e04d645670eb37341990 in 853ms, sequenceid=227, compaction requested=true 2023-04-24 09:00:30,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:30,156 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:30,156 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:30,156 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:30,156 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:30,156 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:30,157 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38806 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:30,157 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/A is initiating minor compaction (all files) 2023-04-24 09:00:30,157 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/A in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,157 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e27287df98ad4c2b923f93854106f6d1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9cbc0f2979b142f890b114f902ed79bb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f576161bd0524272b57e3798de835dda] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=37.9 K 2023-04-24 09:00:30,158 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:30,158 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e27287df98ad4c2b923f93854106f6d1, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682326827339 2023-04-24 09:00:30,158 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/B is initiating minor compaction (all files) 2023-04-24 09:00:30,158 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/B in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,158 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/2ecda413122b4489a6e17f518b7feda5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a769f546d0e34a5db56ae66702fccfb7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/8147be04f4bc4bde8750c829ebe1f691] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=35.6 K 2023-04-24 09:00:30,158 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 9cbc0f2979b142f890b114f902ed79bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682326827462 2023-04-24 09:00:30,158 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 2ecda413122b4489a6e17f518b7feda5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682326827339 2023-04-24 09:00:30,158 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f576161bd0524272b57e3798de835dda, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1682326828668 2023-04-24 09:00:30,158 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a769f546d0e34a5db56ae66702fccfb7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682326827462 2023-04-24 09:00:30,159 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 8147be04f4bc4bde8750c829ebe1f691, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1682326828684 2023-04-24 09:00:30,163 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#A#compaction#372 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:30,164 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#B#compaction#373 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:30,206 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:30,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:30,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:30,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:30,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:30,209 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:30,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:30,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:30,209 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:30,210 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,210 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,220 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:30,220 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:30,220 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,220 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:30,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:30,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:30,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:30,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:30,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:30,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:30,221 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:30,221 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:30,222 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 09:00:30,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:30,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:30,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:30,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:30,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:30,222 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:30,230 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/36cd4b3d75d74732af9b1a12e33ccc49 2023-04-24 09:00:30,241 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/f9cf09d8bfe449779b507f27b1c95493 2023-04-24 09:00:30,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:30,440 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:30,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326830447, max timestamp=1682326830476Num uniques 859; 2023-04-24 09:00:30,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:30,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326830447, max timestamp=1682326830476Num uniques 859; 2023-04-24 09:00:30,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:30,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326830447, max timestamp=1682326830476Num uniques 859; 2023-04-24 09:00:30,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:30,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326830500, max timestamp=1682326830520Num uniques 1118; 2023-04-24 09:00:30,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-24 09:00:30,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326830500, max timestamp=1682326830520Num uniques 1118; 2023-04-24 09:00:30,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326830500, max timestamp=1682326830520Num uniques 1118; 2023-04-24 09:00:30,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-24 09:00:30,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-24 09:00:30,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682326830536, max timestamp=1682326830561Num uniques 1127; 2023-04-24 09:00:30,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-24 09:00:30,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682326830536, max timestamp=1682326830561Num uniques 1127; 2023-04-24 09:00:30,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-24 09:00:30,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1682326830536, max timestamp=1682326830561Num uniques 1127; 2023-04-24 09:00:30,564 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-24 09:00:30,582 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/779a9733c25d4bdbb50dfc9c52f5ca37 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/779a9733c25d4bdbb50dfc9c52f5ca37 2023-04-24 09:00:30,585 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/B of 3695ff097ea6e04d645670eb37341990 into 779a9733c25d4bdbb50dfc9c52f5ca37(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:30,585 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:30,585 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/B, priority=13, startTime=1682326830156; duration=0sec 2023-04-24 09:00:30,585 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:30,585 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:30,588 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:30,588 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/C is initiating minor compaction (all files) 2023-04-24 09:00:30,588 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/C in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,588 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e1b5b4f241604d1b89fcf3b9cf18b26e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/676b2e94075140ed98baef23d6f53b8f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dd87ef75633742089aa2844d24b9354a] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=35.6 K 2023-04-24 09:00:30,588 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e1b5b4f241604d1b89fcf3b9cf18b26e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1682326827339 2023-04-24 09:00:30,589 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 676b2e94075140ed98baef23d6f53b8f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1682326827462 2023-04-24 09:00:30,589 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting dd87ef75633742089aa2844d24b9354a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1682326828684 2023-04-24 09:00:30,593 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#C#compaction#377 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:30,596 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/be9c81b31bb9491faeb4e0bf68ed91f1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/be9c81b31bb9491faeb4e0bf68ed91f1 2023-04-24 09:00:30,600 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/a36f91855efd4e41b35b597f16d069d3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/a36f91855efd4e41b35b597f16d069d3 2023-04-24 09:00:30,600 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/A of 3695ff097ea6e04d645670eb37341990 into be9c81b31bb9491faeb4e0bf68ed91f1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:30,600 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:30,600 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/A, priority=13, startTime=1682326830156; duration=0sec 2023-04-24 09:00:30,600 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:30,605 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/C of 3695ff097ea6e04d645670eb37341990 into a36f91855efd4e41b35b597f16d069d3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:30,605 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:30,605 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/C, priority=13, startTime=1682326830156; duration=0sec 2023-04-24 09:00:30,605 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:30,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:30,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326890611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326830599, max timestamp=1682326830614Num uniques 1136; 2023-04-24 09:00:30,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-24 09:00:30,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326830599, max timestamp=1682326830614Num uniques 1136; 2023-04-24 09:00:30,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682326830599, max timestamp=1682326830614Num uniques 1136; 2023-04-24 09:00:30,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-24 09:00:30,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-24 09:00:30,651 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/d6511b121b204add88b848140b3617a5 2023-04-24 09:00:30,654 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/36cd4b3d75d74732af9b1a12e33ccc49 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/36cd4b3d75d74732af9b1a12e33ccc49 2023-04-24 09:00:30,656 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/36cd4b3d75d74732af9b1a12e33ccc49, entries=100, sequenceid=234, filesize=9.4 K 2023-04-24 09:00:30,657 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/f9cf09d8bfe449779b507f27b1c95493 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f9cf09d8bfe449779b507f27b1c95493 2023-04-24 09:00:30,659 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f9cf09d8bfe449779b507f27b1c95493, entries=100, sequenceid=234, filesize=9.4 K 2023-04-24 09:00:30,662 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/d6511b121b204add88b848140b3617a5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/d6511b121b204add88b848140b3617a5 2023-04-24 09:00:30,665 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/d6511b121b204add88b848140b3617a5, entries=100, sequenceid=234, filesize=9.4 K 2023-04-24 09:00:30,665 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=134.18 KB/137400 for 3695ff097ea6e04d645670eb37341990 in 443ms, sequenceid=234, compaction requested=false 2023-04-24 09:00:30,665 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:30,665 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:30,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:30,669 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,670 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:30,670 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:30,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:30,671 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=140.89 KB heapSize=397.69 KB 2023-04-24 09:00:30,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:30,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:30,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:30,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:30,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:30,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:30,680 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,680 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/2c33f8bf64a147259d4b3133f301c289 2023-04-24 09:00:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,684 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,684 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:30,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:30,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b69322[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:30,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:30,684 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,689 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,689 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:30,689 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:30,689 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:30,689 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,689 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:30,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:30,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326890687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:30,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326890690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,697 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:30,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:30,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:30,699 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:30,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:30,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:30,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:30,699 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:30,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/764cb295cf544258976fed54594cf116 2023-04-24 09:00:30,705 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,705 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,705 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,706 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,707 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:30,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326890717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:30,722 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:30,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:30,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:30,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:30,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:30,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:30,723 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,723 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,723 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,723 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,723 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,723 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,723 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,723 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:30,723 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:30,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:30,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326890794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:30,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326890796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,824 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:30,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:30,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:30,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:30,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:30,826 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:30,826 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:30,826 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:30,826 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:30,826 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,826 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:30,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57162 deadline: 1682326890830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,836 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:30,836 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:30,837 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,837 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:30,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:30,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:30,837 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:30,837 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:30,837 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:30,838 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:30,838 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:30,838 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,838 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:30,838 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:30,838 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:30,838 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:30,838 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:30,838 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,838 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:30,838 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:30,847 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:30,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57bae10a[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:30,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:30,848 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,884 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,885 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:30,885 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,885 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:30,885 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:30,885 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:30,903 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:30,903 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:30,903 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,903 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:30,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:30,905 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:30,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:30,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:30,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:30,905 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:30,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:30,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,915 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:30,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:30,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,922 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:30,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:30,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:30,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:30,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:30,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:30,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:30,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:30,923 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:30,923 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,923 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:30,923 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:30,923 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:30,923 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:30,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:30,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326890924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326890998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326891002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,023 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:31,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:31,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:31,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:31,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:31,026 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:31,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:31,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:31,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:31,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:31,036 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,036 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:31,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:31,037 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,037 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:31,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:31,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:31,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:31,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:31,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:31,038 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:31,038 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:31,038 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:31,038 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:31,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:31,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:31,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:31,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:31,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:31,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:31,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316f8280[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:31,048 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:31,048 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:31,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:31,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:31,055 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:31,055 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:31,055 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,055 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:31,055 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:31,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:31,063 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:31,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:31,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:31,065 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:31,065 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:31,065 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:31,065 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:31,065 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:31,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:31,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:31,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:31,073 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:31,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:31,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:31,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:31,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:31,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:31,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:31,080 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:31,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:31,081 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:31,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:31,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:31,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:31,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:31,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/212e120037d14a8b9cefd8dd1f1c2198 2023-04-24 09:00:31,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/2c33f8bf64a147259d4b3133f301c289 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/2c33f8bf64a147259d4b3133f301c289 2023-04-24 09:00:31,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/2c33f8bf64a147259d4b3133f301c289, entries=250, sequenceid=326, filesize=16.7 K 2023-04-24 09:00:31,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/764cb295cf544258976fed54594cf116 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/764cb295cf544258976fed54594cf116 2023-04-24 09:00:31,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/764cb295cf544258976fed54594cf116, entries=150, sequenceid=326, filesize=11.9 K 2023-04-24 09:00:31,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/212e120037d14a8b9cefd8dd1f1c2198 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/212e120037d14a8b9cefd8dd1f1c2198 2023-04-24 09:00:31,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/212e120037d14a8b9cefd8dd1f1c2198, entries=150, sequenceid=326, filesize=11.9 K 2023-04-24 09:00:31,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~418.27 KB/428304, currentSize=33.54 KB/34350 for 3695ff097ea6e04d645670eb37341990 in 464ms, sequenceid=326, compaction requested=true 2023-04-24 09:00:31,135 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:31,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:31,138 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:31,138 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:31,138 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:31,138 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:31,138 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39154 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:31,138 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:31,138 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/A is initiating minor compaction (all files) 2023-04-24 09:00:31,139 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/B is initiating minor compaction (all files) 2023-04-24 09:00:31,139 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/A in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:31,139 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/B in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:31,139 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/779a9733c25d4bdbb50dfc9c52f5ca37, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f9cf09d8bfe449779b507f27b1c95493, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/764cb295cf544258976fed54594cf116] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=33.5 K 2023-04-24 09:00:31,139 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/be9c81b31bb9491faeb4e0bf68ed91f1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/36cd4b3d75d74732af9b1a12e33ccc49, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/2c33f8bf64a147259d4b3133f301c289] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=38.2 K 2023-04-24 09:00:31,139 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 779a9733c25d4bdbb50dfc9c52f5ca37, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1682326828684 2023-04-24 09:00:31,139 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting be9c81b31bb9491faeb4e0bf68ed91f1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1682326828684 2023-04-24 09:00:31,139 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f9cf09d8bfe449779b507f27b1c95493, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682326829311 2023-04-24 09:00:31,139 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 36cd4b3d75d74732af9b1a12e33ccc49, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682326829311 2023-04-24 09:00:31,139 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 764cb295cf544258976fed54594cf116, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682326830656 2023-04-24 09:00:31,139 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 2c33f8bf64a147259d4b3133f301c289, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682326830651 2023-04-24 09:00:31,145 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#B#compaction#381 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:31,146 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#A#compaction#382 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:31,159 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/c83f14ff3d2a4dcfa5470c6e99b2d6d6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/c83f14ff3d2a4dcfa5470c6e99b2d6d6 2023-04-24 09:00:31,163 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/B of 3695ff097ea6e04d645670eb37341990 into c83f14ff3d2a4dcfa5470c6e99b2d6d6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:31,163 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:31,163 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/B, priority=13, startTime=1682326831138; duration=0sec 2023-04-24 09:00:31,163 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:31,163 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:31,164 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:31,164 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/C is initiating minor compaction (all files) 2023-04-24 09:00:31,164 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/C in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:31,164 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/a36f91855efd4e41b35b597f16d069d3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/d6511b121b204add88b848140b3617a5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/212e120037d14a8b9cefd8dd1f1c2198] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=33.5 K 2023-04-24 09:00:31,164 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a36f91855efd4e41b35b597f16d069d3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1682326828684 2023-04-24 09:00:31,164 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d6511b121b204add88b848140b3617a5, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1682326829311 2023-04-24 09:00:31,165 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 212e120037d14a8b9cefd8dd1f1c2198, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682326830656 2023-04-24 09:00:31,171 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#C#compaction#383 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:31,181 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:31,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:31,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:31,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:31,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:31,183 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:31,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:31,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:31,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:31,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:31,188 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,188 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:31,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:31,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:31,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:31,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:31,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:31,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:31,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:31,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:31,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:31,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:31,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:31,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:31,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:31,190 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 09:00:31,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:31,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:31,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:31,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:31,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:31,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:31,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:31,237 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:31,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326831293, max timestamp=1682326831312Num uniques 859; 2023-04-24 09:00:31,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326831293, max timestamp=1682326831312Num uniques 859; 2023-04-24 09:00:31,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:31,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:31,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326831293, max timestamp=1682326831312Num uniques 859; 2023-04-24 09:00:31,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:31,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326831336, max timestamp=1682326831355Num uniques 1018; 2023-04-24 09:00:31,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326831336, max timestamp=1682326831355Num uniques 1018; 2023-04-24 09:00:31,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-24 09:00:31,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682326831336, max timestamp=1682326831355Num uniques 1018; 2023-04-24 09:00:31,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-24 09:00:31,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-24 09:00:31,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326891385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326891393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326891393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326891494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326891500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326891500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,559 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/154c271f057b4663982bd6f9cc786222 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/154c271f057b4663982bd6f9cc786222 2023-04-24 09:00:31,564 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/A of 3695ff097ea6e04d645670eb37341990 into 154c271f057b4663982bd6f9cc786222(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:31,564 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:31,564 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/A, priority=13, startTime=1682326831135; duration=0sec 2023-04-24 09:00:31,564 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:31,580 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/febc176f1fd64f2b8f43af638eadde81 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/febc176f1fd64f2b8f43af638eadde81 2023-04-24 09:00:31,583 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/C of 3695ff097ea6e04d645670eb37341990 into febc176f1fd64f2b8f43af638eadde81(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:31,583 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:31,584 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/C, priority=13, startTime=1682326831138; duration=0sec 2023-04-24 09:00:31,584 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:31,597 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/76788337b2e8484cb4f4eaf626b173d6 2023-04-24 09:00:31,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326891699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326891705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:31,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:31,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326891705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,004 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/ec45c2a95b3e4579b1716604a5394ca5 2023-04-24 09:00:32,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:32,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326892003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:32,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326892008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:32,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326892008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:32,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:57192 deadline: 1682326892194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,198 DEBUG [Thread-2662] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:32,413 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/1d33ac8b663b433e93d56f8ad25e3891 2023-04-24 09:00:32,416 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/76788337b2e8484cb4f4eaf626b173d6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/76788337b2e8484cb4f4eaf626b173d6 2023-04-24 09:00:32,422 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/76788337b2e8484cb4f4eaf626b173d6, entries=150, sequenceid=335, filesize=11.9 K 2023-04-24 09:00:32,422 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/ec45c2a95b3e4579b1716604a5394ca5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ec45c2a95b3e4579b1716604a5394ca5 2023-04-24 09:00:32,425 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ec45c2a95b3e4579b1716604a5394ca5, entries=150, sequenceid=335, filesize=11.9 K 2023-04-24 09:00:32,426 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/1d33ac8b663b433e93d56f8ad25e3891 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d33ac8b663b433e93d56f8ad25e3891 2023-04-24 09:00:32,428 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d33ac8b663b433e93d56f8ad25e3891, entries=150, sequenceid=335, filesize=11.9 K 2023-04-24 09:00:32,429 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 3695ff097ea6e04d645670eb37341990 in 1239ms, sequenceid=335, compaction requested=false 2023-04-24 09:00:32,429 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:32,429 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:32,429 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:32,429 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:32,429 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,429 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:32,429 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:32,438 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,439 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,439 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,439 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:32,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc7a9e2[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:32,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:32,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,447 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:32,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:32,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:32,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:32,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:32,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:32,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:32,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:32,457 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:32,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:32,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:32,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:32,457 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:32,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:32,464 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,465 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:32,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:32,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:32,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:32,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:32,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:32,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,472 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:32,472 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:32,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:32,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:32,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,473 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682326831378, max timestamp=1682326832515Num uniques 1127; 2023-04-24 09:00:32,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:32,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682326831378, max timestamp=1682326832515Num uniques 1127; 2023-04-24 09:00:32,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 09:00:32,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682326831378, max timestamp=1682326832515Num uniques 1127; 2023-04-24 09:00:32,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 09:00:32,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-24 09:00:32,518 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=111.82 KB heapSize=311.67 KB 2023-04-24 09:00:32,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:32,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:32,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:32,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:32,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:32,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:32,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326832543, max timestamp=1682326832558Num uniques 859; 2023-04-24 09:00:32,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:32,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326832543, max timestamp=1682326832558Num uniques 859; 2023-04-24 09:00:32,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326832543, max timestamp=1682326832558Num uniques 859; 2023-04-24 09:00:32,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:32,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:32,572 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:32,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:32,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:32,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:32,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:32,581 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:32,581 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:32,581 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:32,581 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:32,581 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,582 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:32,588 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:32,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:32,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:32,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:32,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:32,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:32,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:32,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:32,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:32,596 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:32,596 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:32,597 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:32,597 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:32,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:32,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:32,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:32,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:32,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,605 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,606 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,606 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:32,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd914db[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:32,608 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:32,608 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,613 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,614 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:32,614 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:32,614 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:32,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:32,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326832602, max timestamp=1682326832618Num uniques 1018; 2023-04-24 09:00:32,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 09:00:32,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326832602, max timestamp=1682326832618Num uniques 1018; 2023-04-24 09:00:32,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682326832602, max timestamp=1682326832618Num uniques 1018; 2023-04-24 09:00:32,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 09:00:32,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-24 09:00:32,641 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:32,641 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,641 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:32,642 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:32,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:32,645 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:32,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:32,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:32,645 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:32,645 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:32,655 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,655 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:32,655 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,657 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,658 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,663 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:32,663 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,664 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,664 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,664 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,664 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,664 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,664 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:32,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:32,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:32,664 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:32,664 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:32,664 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:32,664 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:32,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8093, totalHeapSize=18648, min timestamp=1682326832668, max timestamp=1682326832692Num uniques 1077; 2023-04-24 09:00:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8093, totalHeapSize=18648, min timestamp=1682326832668, max timestamp=1682326832692Num uniques 1077; 2023-04-24 09:00:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=49313, new segment data size=8093, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-24 09:00:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1682326832668, max timestamp=1682326832692Num uniques 1077; 2023-04-24 09:00:32,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=8092, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-24 09:00:32,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=49313, new segment data size=8093, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-24 09:00:32,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682326832692, max timestamp=1682326832750Num uniques 1036; 2023-04-24 09:00:32,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-24 09:00:32,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682326832692, max timestamp=1682326832750Num uniques 1036; 2023-04-24 09:00:32,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-24 09:00:32,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682326832692, max timestamp=1682326832750Num uniques 1036; 2023-04-24 09:00:32,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-24 09:00:32,764 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:32,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:32,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:32,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:32,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:32,768 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:32,768 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:32,768 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:32,768 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:32,768 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,768 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,778 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:32,778 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:32,778 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,778 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:32,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:32,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:32,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:32,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:32,779 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:32,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:32,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:32,780 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:32,780 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:32,780 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:32,780 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:32,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:32,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:32,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:32,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:32,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,788 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:32,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3deb00db[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:32,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:32,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:32,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:32,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:32,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:32,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:32,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:32,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:32,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:32,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:32,807 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:32,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:32,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:32,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:32,807 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:32,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:32,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326892798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326832787, max timestamp=1682326832810Num uniques 1145; 2023-04-24 09:00:32,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-24 09:00:32,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326832787, max timestamp=1682326832810Num uniques 1145; 2023-04-24 09:00:32,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-24 09:00:32,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682326832787, max timestamp=1682326832810Num uniques 1145; 2023-04-24 09:00:32,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-24 09:00:32,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:32,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,814 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,815 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:32,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:32,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:32,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:32,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:32,822 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:32,822 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:32,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:32,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,823 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:32,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:32,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682326832861, max timestamp=1682326832875Num uniques 1104; 2023-04-24 09:00:32,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-24 09:00:32,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682326832861, max timestamp=1682326832875Num uniques 1104; 2023-04-24 09:00:32,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-24 09:00:32,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1682326832861, max timestamp=1682326832875Num uniques 1104; 2023-04-24 09:00:32,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-24 09:00:32,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682326832902, max timestamp=1682326832913Num uniques 1113; 2023-04-24 09:00:32,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-24 09:00:32,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:32,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326892912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682326832902, max timestamp=1682326832913Num uniques 1113; 2023-04-24 09:00:32,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-24 09:00:32,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1682326832902, max timestamp=1682326832913Num uniques 1113; 2023-04-24 09:00:32,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-24 09:00:32,922 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:32,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:32,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:32,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:32,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:32,925 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:32,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:32,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:32,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:32,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/891a67b7d6694ab9981c30e9db651970 2023-04-24 09:00:32,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:32,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:32,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:32,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:32,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:32,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:32,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:32,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:32,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:32,932 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:32,932 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:32,932 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:32,932 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:32,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:32,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:32,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:32,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:32,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,939 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,939 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:32,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ddfe64[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:32,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:32,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:32,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:32,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:32,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:32,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:32,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:32,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:32,958 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:32,958 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:32,958 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:32,958 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:32,958 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:32,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326832935, max timestamp=1682326832959Num uniques 1122; 2023-04-24 09:00:32,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326832935, max timestamp=1682326832959Num uniques 1122; 2023-04-24 09:00:32,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-24 09:00:32,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-24 09:00:32,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682326832935, max timestamp=1682326832959Num uniques 1122; 2023-04-24 09:00:32,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-24 09:00:32,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:32,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:32,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:32,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:32,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:32,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:32,972 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:32,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:32,972 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:32,972 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:32,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682326832997, max timestamp=1682326833010Num uniques 1031; 2023-04-24 09:00:33,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-24 09:00:33,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682326832997, max timestamp=1682326833010Num uniques 1031; 2023-04-24 09:00:33,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682326832997, max timestamp=1682326833010Num uniques 1031; 2023-04-24 09:00:33,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-24 09:00:33,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-24 09:00:33,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682326833026, max timestamp=1682326833046Num uniques 1040; 2023-04-24 09:00:33,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-24 09:00:33,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682326833026, max timestamp=1682326833046Num uniques 1040; 2023-04-24 09:00:33,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-24 09:00:33,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1682326833026, max timestamp=1682326833046Num uniques 1040; 2023-04-24 09:00:33,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-24 09:00:33,073 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:33,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:33,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:33,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:33,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:33,075 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:33,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:33,076 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:33,076 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,076 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:33,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:33,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:33,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:33,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:33,081 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:33,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:33,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:33,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:33,082 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:33,088 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:33,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e4c124[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:33,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:33,090 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,097 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,097 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,097 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,097 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:33,097 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:33,097 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:33,105 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:33,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,105 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:33,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:33,107 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:33,107 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:33,107 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:33,107 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:33,107 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:33,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,114 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,115 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326893110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682326833096, max timestamp=1682326833115Num uniques 1149; 2023-04-24 09:00:33,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-24 09:00:33,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682326833096, max timestamp=1682326833115Num uniques 1149; 2023-04-24 09:00:33,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1682326833096, max timestamp=1682326833115Num uniques 1149; 2023-04-24 09:00:33,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-24 09:00:33,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-24 09:00:33,142 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,142 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:33,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:33,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,142 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,142 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,142 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:33,142 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,142 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:33,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:33,142 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,142 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,142 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,143 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,143 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326833164, max timestamp=1682326833178Num uniques 1108; 2023-04-24 09:00:33,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 09:00:33,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326833164, max timestamp=1682326833178Num uniques 1108; 2023-04-24 09:00:33,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 09:00:33,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1682326833164, max timestamp=1682326833178Num uniques 1108; 2023-04-24 09:00:33,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-24 09:00:33,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682326833219, max timestamp=1682326833232Num uniques 1067; 2023-04-24 09:00:33,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-24 09:00:33,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682326833219, max timestamp=1682326833232Num uniques 1067; 2023-04-24 09:00:33,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1682326833219, max timestamp=1682326833232Num uniques 1067; 2023-04-24 09:00:33,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-24 09:00:33,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326893228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-24 09:00:33,242 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:33,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:33,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:33,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:33,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:33,245 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:33,245 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:33,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:33,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,255 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,256 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,256 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:33,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:33,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:33,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:33,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:33,256 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:33,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:33,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:33,257 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,257 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:33,257 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:33,257 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:33,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:33,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:33,257 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:33,263 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,264 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:33,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:33,266 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:33,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ee6a0f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:33,266 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,272 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:33,272 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:33,272 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:33,272 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,272 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:33,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:33,280 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,280 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:33,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:33,282 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:33,282 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:33,282 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:33,282 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:33,282 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:33,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682326833270, max timestamp=1682326833286Num uniques 1076; 2023-04-24 09:00:33,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-24 09:00:33,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682326833270, max timestamp=1682326833286Num uniques 1076; 2023-04-24 09:00:33,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-24 09:00:33,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1682326833270, max timestamp=1682326833286Num uniques 1076; 2023-04-24 09:00:33,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-24 09:00:33,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,297 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,298 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:33,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:33,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,314 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,314 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:33,314 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,314 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:33,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:33,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,314 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,314 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,314 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,314 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,314 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682326833303, max timestamp=1682326833330Num uniques 1035; 2023-04-24 09:00:33,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-24 09:00:33,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682326833303, max timestamp=1682326833330Num uniques 1035; 2023-04-24 09:00:33,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-24 09:00:33,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682326833303, max timestamp=1682326833330Num uniques 1035; 2023-04-24 09:00:33,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-24 09:00:33,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/f14e273f886641c4880971cbb7edd984 2023-04-24 09:00:33,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/15f4b11bf3164b6d8db3815e4c20c841 2023-04-24 09:00:33,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/891a67b7d6694ab9981c30e9db651970 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/891a67b7d6694ab9981c30e9db651970 2023-04-24 09:00:33,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/891a67b7d6694ab9981c30e9db651970, entries=250, sequenceid=393, filesize=16.7 K 2023-04-24 09:00:33,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/f14e273f886641c4880971cbb7edd984 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f14e273f886641c4880971cbb7edd984 2023-04-24 09:00:33,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f14e273f886641c4880971cbb7edd984, entries=150, sequenceid=393, filesize=11.9 K 2023-04-24 09:00:33,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/15f4b11bf3164b6d8db3815e4c20c841 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/15f4b11bf3164b6d8db3815e4c20c841 2023-04-24 09:00:33,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/15f4b11bf3164b6d8db3815e4c20c841, entries=150, sequenceid=393, filesize=11.9 K 2023-04-24 09:00:33,374 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~84.42 KB/86448, currentSize=120.76 KB/123660 for 3695ff097ea6e04d645670eb37341990 in 856ms, sequenceid=393, compaction requested=true 2023-04-24 09:00:33,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:33,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:33,375 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:33,375 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:33,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:33,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:33,375 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41950 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:33,375 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/A is initiating minor compaction (all files) 2023-04-24 09:00:33,376 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/A in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,376 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/154c271f057b4663982bd6f9cc786222, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/76788337b2e8484cb4f4eaf626b173d6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/891a67b7d6694ab9981c30e9db651970] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=41.0 K 2023-04-24 09:00:33,376 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:33,376 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/B is initiating minor compaction (all files) 2023-04-24 09:00:33,376 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/B in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,376 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/c83f14ff3d2a4dcfa5470c6e99b2d6d6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ec45c2a95b3e4579b1716604a5394ca5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f14e273f886641c4880971cbb7edd984] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=36.2 K 2023-04-24 09:00:33,376 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c83f14ff3d2a4dcfa5470c6e99b2d6d6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682326830656 2023-04-24 09:00:33,376 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 154c271f057b4663982bd6f9cc786222, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682326830656 2023-04-24 09:00:33,377 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting ec45c2a95b3e4579b1716604a5394ca5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682326830677 2023-04-24 09:00:33,377 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 76788337b2e8484cb4f4eaf626b173d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682326830677 2023-04-24 09:00:33,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:33,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=127.47 KB heapSize=362.63 KB 2023-04-24 09:00:33,378 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f14e273f886641c4880971cbb7edd984, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682326831378 2023-04-24 09:00:33,378 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 891a67b7d6694ab9981c30e9db651970, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682326831378 2023-04-24 09:00:33,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:33,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:33,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:33,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:33,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:33,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:33,383 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#A#compaction#391 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:33,391 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#B#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:33,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326893393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326893393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326893394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326893395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,414 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:33,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:33,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:33,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:33,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:33,424 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:33,424 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,424 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:33,424 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:33,424 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,424 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,438 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,438 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:33,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:33,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:33,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:33,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,441 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:33,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:33,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:33,444 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:33,444 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:33,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,456 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,456 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:33,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51664991[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:33,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:33,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:33,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:33,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:33,472 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:33,488 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:33,488 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,488 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,489 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:33,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:33,490 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:33,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:33,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:33,491 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:33,491 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:33,505 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,505 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,505 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,506 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,506 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326893503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326893504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326893504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326893504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,551 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:33,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:33,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:33,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,551 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,552 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,552 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,552 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,552 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:33,552 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:33,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,652 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:33,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:33,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:33,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:33,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:33,660 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:33,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:33,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:33,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,660 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,670 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,670 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,670 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,670 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:33,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:33,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:33,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:33,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,672 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:33,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:33,676 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,676 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:33,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:33,676 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:33,676 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:33,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:33,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:33,676 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:33,686 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,686 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,687 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:33,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed7a307[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:33,690 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:33,690 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,698 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:33,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:33,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:33,698 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,698 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:33,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:33,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:33,708 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:33,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:33,710 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:33,710 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:33,710 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:33,710 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:33,711 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:33,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326893709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326893711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326893713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:33,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326893713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,722 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,724 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,751 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,751 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,751 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,751 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:33,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:33,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,751 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:33,751 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,752 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:33,752 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:33,752 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,752 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,752 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,752 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/d13c903a32f5474ab9afd92a35733c1a 2023-04-24 09:00:33,816 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/beea66d1f4cd49b6b4a12408efcef73d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/beea66d1f4cd49b6b4a12408efcef73d 2023-04-24 09:00:33,819 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/A of 3695ff097ea6e04d645670eb37341990 into beea66d1f4cd49b6b4a12408efcef73d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:33,820 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:33,820 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/A, priority=13, startTime=1682326833375; duration=0sec 2023-04-24 09:00:33,820 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:33,820 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:33,822 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:33,822 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/C is initiating minor compaction (all files) 2023-04-24 09:00:33,822 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/C in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,822 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/febc176f1fd64f2b8f43af638eadde81, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d33ac8b663b433e93d56f8ad25e3891, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/15f4b11bf3164b6d8db3815e4c20c841] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=36.2 K 2023-04-24 09:00:33,822 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting febc176f1fd64f2b8f43af638eadde81, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1682326830656 2023-04-24 09:00:33,823 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1d33ac8b663b433e93d56f8ad25e3891, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1682326830677 2023-04-24 09:00:33,823 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 15f4b11bf3164b6d8db3815e4c20c841, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682326831378 2023-04-24 09:00:33,827 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/737ebc246a824788a5a6f9d9bef705cc as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/737ebc246a824788a5a6f9d9bef705cc 2023-04-24 09:00:33,827 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#C#compaction#394 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:33,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/273880aeb5954ba8bf01efc851bd58fb 2023-04-24 09:00:33,831 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/B of 3695ff097ea6e04d645670eb37341990 into 737ebc246a824788a5a6f9d9bef705cc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:33,831 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:33,831 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/B, priority=13, startTime=1682326833375; duration=0sec 2023-04-24 09:00:33,831 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:33,852 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:33,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:33,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:33,855 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:33,855 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,855 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:33,855 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:33,855 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,855 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,863 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,863 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,864 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,864 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:33,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:33,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:33,865 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,865 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:33,865 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:33,865 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:33,865 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:33,872 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,872 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,872 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,872 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:33,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:33,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29dda618[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:33,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,880 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,880 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,880 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:33,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:33,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:33,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:33,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:33,888 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,889 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,889 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:33,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:33,890 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:33,890 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:33,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:33,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:33,891 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:33,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:33,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:33,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,899 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:33,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:33,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:33,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:33,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:33,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:33,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:33,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:33,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:33,906 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:33,906 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:33,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:33,906 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:33,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:34,006 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:34,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:34,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:34,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:34,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:34,008 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:34,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:34,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:34,008 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:34,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,009 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,013 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,013 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:34,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:34,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:34,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:34,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:34,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:34,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:34,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:34,015 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:34,015 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:34,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,022 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:34,022 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:34,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326894017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,022 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:34,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea41f3e[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:34,023 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:34,023 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326894021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326894021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326894022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:34,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:34,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:34,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:34,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:34,038 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,038 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:34,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:34,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:34,040 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:34,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:34,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:34,040 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:34,040 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:34,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:34,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:34,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:34,047 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,048 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:34,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:34,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:34,055 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:34,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:34,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:34,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:34,055 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:34,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:34,055 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:34,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,056 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,156 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:34,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:34,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:34,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:34,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:34,158 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:34,158 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:34,158 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:34,158 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:34,159 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,159 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,163 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:34,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:34,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:34,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:34,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:34,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:34,165 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:34,165 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:34,165 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:34,165 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:34,165 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:34,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,172 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:34,172 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,172 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:34,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:34,174 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:34,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d6df9ba[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:34,174 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,180 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,180 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:34,180 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,180 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:34,180 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:34,181 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:34,188 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:34,188 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:34,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:34,188 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:34,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:34,191 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:34,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:34,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:34,191 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:34,191 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:34,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:34,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:34,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:34,197 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,199 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:34,205 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:34,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:34,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:34,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:34,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:34,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:34,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:34,206 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:34,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:34,206 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:34,206 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,237 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/2b2c7099425549ef9ef28a4349690136 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2b2c7099425549ef9ef28a4349690136 2023-04-24 09:00:34,240 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/C of 3695ff097ea6e04d645670eb37341990 into 2b2c7099425549ef9ef28a4349690136(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:34,240 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:34,240 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/C, priority=13, startTime=1682326833375; duration=0sec 2023-04-24 09:00:34,240 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:34,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/4bd256099ab2429fa06ce7f9166538e6 2023-04-24 09:00:34,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/d13c903a32f5474ab9afd92a35733c1a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/d13c903a32f5474ab9afd92a35733c1a 2023-04-24 09:00:34,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/d13c903a32f5474ab9afd92a35733c1a, entries=250, sequenceid=670, filesize=16.7 K 2023-04-24 09:00:34,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/273880aeb5954ba8bf01efc851bd58fb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/273880aeb5954ba8bf01efc851bd58fb 2023-04-24 09:00:34,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/273880aeb5954ba8bf01efc851bd58fb, entries=150, sequenceid=670, filesize=11.9 K 2023-04-24 09:00:34,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/4bd256099ab2429fa06ce7f9166538e6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/4bd256099ab2429fa06ce7f9166538e6 2023-04-24 09:00:34,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/4bd256099ab2429fa06ce7f9166538e6, entries=150, sequenceid=670, filesize=11.9 K 2023-04-24 09:00:34,256 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~382.50 KB/391680, currentSize=46.96 KB/48090 for 3695ff097ea6e04d645670eb37341990 in 879ms, sequenceid=670, compaction requested=false 2023-04-24 09:00:34,256 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:34,306 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:34,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:34,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:34,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:34,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:34,309 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:34,309 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:34,309 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:34,309 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:34,309 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,309 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,339 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:34,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:34,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:34,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:34,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:34,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:34,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:34,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:34,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:34,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:34,344 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:34,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:34,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:34,344 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 09:00:34,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:34,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:34,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:34,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:34,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:34,344 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:34,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:34,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:34,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326834547, max timestamp=1682326834559Num uniques 859; 2023-04-24 09:00:34,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:34,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326834547, max timestamp=1682326834559Num uniques 859; 2023-04-24 09:00:34,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:34,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326834547, max timestamp=1682326834559Num uniques 859; 2023-04-24 09:00:34,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:34,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326894578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326894582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326894583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326894585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326894686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326894686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326894691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326894694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,762 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/9f62d061fba2476d9f0e6fd3cbcde1bb 2023-04-24 09:00:34,770 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/6a5e842addf44a6f99fbbf6298eb5827 2023-04-24 09:00:34,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326894892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326894893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326894893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:34,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:34,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326894899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,180 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/6d79a7920d3b4a4e872a85feb704eb0e 2023-04-24 09:00:35,183 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/9f62d061fba2476d9f0e6fd3cbcde1bb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9f62d061fba2476d9f0e6fd3cbcde1bb 2023-04-24 09:00:35,186 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9f62d061fba2476d9f0e6fd3cbcde1bb, entries=150, sequenceid=683, filesize=11.9 K 2023-04-24 09:00:35,187 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/6a5e842addf44a6f99fbbf6298eb5827 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/6a5e842addf44a6f99fbbf6298eb5827 2023-04-24 09:00:35,189 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/6a5e842addf44a6f99fbbf6298eb5827, entries=150, sequenceid=683, filesize=11.9 K 2023-04-24 09:00:35,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/6d79a7920d3b4a4e872a85feb704eb0e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6d79a7920d3b4a4e872a85feb704eb0e 2023-04-24 09:00:35,192 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6d79a7920d3b4a4e872a85feb704eb0e, entries=150, sequenceid=683, filesize=11.9 K 2023-04-24 09:00:35,193 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3695ff097ea6e04d645670eb37341990 in 849ms, sequenceid=683, compaction requested=true 2023-04-24 09:00:35,193 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:35,193 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,193 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:35,193 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:35,193 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,193 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:35,193 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:35,203 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,203 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,203 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,203 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:35,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e51bc8b[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:35,204 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:35,204 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:35,208 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-24 09:00:35,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:35,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:35,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:35,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:35,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:35,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:35,213 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,214 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:35,214 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:35,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=720 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/03c1dba1ff464b3cb4f85db27a1ae70f 2023-04-24 09:00:35,214 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:35,214 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,214 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:35,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326895213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326895215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:35,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,222 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:35,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:35,225 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:35,225 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:35,225 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:35,225 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:35,225 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:35,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326895219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326895220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,231 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,239 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:35,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,239 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:35,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,239 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:35,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326895321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326895321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326895328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326895329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,339 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:35,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:35,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:35,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:35,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:35,342 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:35,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:35,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:35,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,347 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,347 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,347 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,347 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:35,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:35,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:35,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:35,348 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:35,348 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,348 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:35,348 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:35,348 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:35,348 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,348 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:35,349 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:35,349 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,349 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:35,349 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:35,349 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,349 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:35,349 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:35,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,355 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,355 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,355 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:35,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:35,357 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:35,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3af8383f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:35,357 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,363 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,364 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:35,364 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:35,364 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:35,364 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,364 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:35,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:35,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,372 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,372 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:35,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:35,375 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:35,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:35,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:35,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:35,375 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:35,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,388 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,388 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:35,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:35,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,389 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:35,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:35,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,389 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:35,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,389 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,489 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:35,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:35,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:35,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:35,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:35,492 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:35,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:35,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:35,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,492 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,497 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,497 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,497 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,497 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:35,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:35,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:35,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:35,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:35,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:35,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:35,498 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,498 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:35,498 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:35,498 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:35,498 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:35,499 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,499 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:35,499 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:35,505 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,505 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,505 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,505 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:35,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74eb8db1[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:35,506 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:35,506 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,513 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,514 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:35,514 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:35,514 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:35,514 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:35,522 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,522 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:35,522 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,522 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:35,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:35,524 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:35,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:35,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:35,524 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:35,524 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:35,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326895527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326895528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,530 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,530 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,530 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,530 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,531 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326895535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326895535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,538 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,538 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:35,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:35,538 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,538 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,539 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,539 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:35,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:35,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,539 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,539 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:35,539 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,539 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,539 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,539 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=720 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/d3d4ef2bd4ce4464bc1d8f93fb93ca03 2023-04-24 09:00:35,639 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:35,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:35,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:35,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:35,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:35,641 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:35,641 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,641 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:35,641 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:35,642 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,642 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:35,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:35,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:35,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:35,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:35,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:35,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:35,679 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,679 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,679 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,679 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:35,680 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:35,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a90abf[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:35,680 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,689 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:35,689 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,689 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:35,689 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:35,689 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:35,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:35,697 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:35,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:35,698 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:35,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:35,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:35,699 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:35,699 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:35,705 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,705 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,705 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,705 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,706 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:35,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:35,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,714 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,714 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,713 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,714 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:35,714 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,714 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:35,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,714 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:35,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,814 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:35,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:35,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:35,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:35,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:35,817 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:35,817 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,817 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:35,817 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:35,817 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,817 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,822 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:35,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:35,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:35,823 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,823 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:35,823 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:35,823 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:35,823 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:35,830 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,830 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,830 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,830 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:35,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ef4905[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:35,832 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:35,832 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326895831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326895831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,839 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,839 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:35,839 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:35,839 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:35,839 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:35,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326895837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:35,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326895839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,847 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,847 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:35,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:35,849 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:35,849 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:35,849 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:35,849 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:35,849 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:35,855 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,855 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,855 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,855 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,856 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,863 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,863 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,863 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,863 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:35,863 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,863 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:35,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:35,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:35,864 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,864 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,864 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,864 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:35,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:35,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:35,864 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:35,864 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:35,964 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:35,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:35,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:35,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:35,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:35,967 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:35,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:35,967 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:35,968 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,968 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,978 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:35,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:35,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:35,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,978 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:35,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:35,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:35,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:35,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:35,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:35,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:35,979 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:35,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:35,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:35,980 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:35,980 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:35,988 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,988 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:35,989 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:35,989 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:35,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:35,990 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:35,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e2dbc47[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:35,990 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,997 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:35,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:35,997 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:35,997 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:35,997 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:35,997 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:36,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:36,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:36,005 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,005 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:36,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:36,007 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:36,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:36,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:36,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:36,007 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:36,007 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:36,013 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,013 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:36,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,013 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:36,014 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:36,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:36,015 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:36,022 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:36,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:36,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:36,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:36,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:36,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:36,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:36,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:36,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:36,022 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:36,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,022 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:36,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:36,023 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:36,023 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=720 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/233b1ac3d35442749a7901a609a847d5 2023-04-24 09:00:36,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/03c1dba1ff464b3cb4f85db27a1ae70f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/03c1dba1ff464b3cb4f85db27a1ae70f 2023-04-24 09:00:36,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/03c1dba1ff464b3cb4f85db27a1ae70f, entries=200, sequenceid=720, filesize=14.3 K 2023-04-24 09:00:36,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/d3d4ef2bd4ce4464bc1d8f93fb93ca03 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/d3d4ef2bd4ce4464bc1d8f93fb93ca03 2023-04-24 09:00:36,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/d3d4ef2bd4ce4464bc1d8f93fb93ca03, entries=150, sequenceid=720, filesize=11.9 K 2023-04-24 09:00:36,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/233b1ac3d35442749a7901a609a847d5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/233b1ac3d35442749a7901a609a847d5 2023-04-24 09:00:36,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/233b1ac3d35442749a7901a609a847d5, entries=150, sequenceid=720, filesize=11.9 K 2023-04-24 09:00:36,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 3695ff097ea6e04d645670eb37341990 in 835ms, sequenceid=720, compaction requested=true 2023-04-24 09:00:36,043 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:36,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:36,043 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:36,043 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:36,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:36,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:36,044 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56678 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:36,044 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:36,044 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/A is initiating minor compaction (all files) 2023-04-24 09:00:36,044 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/B is initiating minor compaction (all files) 2023-04-24 09:00:36,044 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/A in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:36,044 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/B in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:36,044 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/beea66d1f4cd49b6b4a12408efcef73d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/d13c903a32f5474ab9afd92a35733c1a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9f62d061fba2476d9f0e6fd3cbcde1bb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/03c1dba1ff464b3cb4f85db27a1ae70f] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=55.3 K 2023-04-24 09:00:36,044 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/737ebc246a824788a5a6f9d9bef705cc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/273880aeb5954ba8bf01efc851bd58fb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/6a5e842addf44a6f99fbbf6298eb5827, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/d3d4ef2bd4ce4464bc1d8f93fb93ca03] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=48.2 K 2023-04-24 09:00:36,045 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting beea66d1f4cd49b6b4a12408efcef73d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682326831378 2023-04-24 09:00:36,045 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 737ebc246a824788a5a6f9d9bef705cc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682326831378 2023-04-24 09:00:36,045 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting d13c903a32f5474ab9afd92a35733c1a, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682326833354 2023-04-24 09:00:36,045 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 273880aeb5954ba8bf01efc851bd58fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682326833354 2023-04-24 09:00:36,045 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 9f62d061fba2476d9f0e6fd3cbcde1bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682326833391 2023-04-24 09:00:36,045 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 6a5e842addf44a6f99fbbf6298eb5827, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682326833391 2023-04-24 09:00:36,045 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 03c1dba1ff464b3cb4f85db27a1ae70f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=720, earliestPutTs=1682326834577 2023-04-24 09:00:36,045 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d3d4ef2bd4ce4464bc1d8f93fb93ca03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=720, earliestPutTs=1682326834582 2023-04-24 09:00:36,051 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#B#compaction#402 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:36,051 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#A#compaction#403 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:36,061 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/4d086450cef24b1e88403e05191e4804 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4d086450cef24b1e88403e05191e4804 2023-04-24 09:00:36,064 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3695ff097ea6e04d645670eb37341990/A of 3695ff097ea6e04d645670eb37341990 into 4d086450cef24b1e88403e05191e4804(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:36,064 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:36,064 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/A, priority=12, startTime=1682326836043; duration=0sec 2023-04-24 09:00:36,064 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:36,064 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:36,065 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:36,065 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/C is initiating minor compaction (all files) 2023-04-24 09:00:36,065 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/C in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:36,065 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2b2c7099425549ef9ef28a4349690136, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/4bd256099ab2429fa06ce7f9166538e6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6d79a7920d3b4a4e872a85feb704eb0e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/233b1ac3d35442749a7901a609a847d5] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=48.2 K 2023-04-24 09:00:36,066 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 2b2c7099425549ef9ef28a4349690136, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1682326831378 2023-04-24 09:00:36,066 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 4bd256099ab2429fa06ce7f9166538e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1682326833354 2023-04-24 09:00:36,066 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 6d79a7920d3b4a4e872a85feb704eb0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1682326833391 2023-04-24 09:00:36,066 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 233b1ac3d35442749a7901a609a847d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=720, earliestPutTs=1682326834582 2023-04-24 09:00:36,071 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#C#compaction#404 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:36,077 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/6a11eec6ef594280af07e7c0bce3d0d3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6a11eec6ef594280af07e7c0bce3d0d3 2023-04-24 09:00:36,080 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3695ff097ea6e04d645670eb37341990/C of 3695ff097ea6e04d645670eb37341990 into 6a11eec6ef594280af07e7c0bce3d0d3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:36,080 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:36,080 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/C, priority=12, startTime=1682326836043; duration=0sec 2023-04-24 09:00:36,080 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:36,123 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:36,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:36,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:36,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:36,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:36,126 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:36,126 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:36,126 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:36,126 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:36,126 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,126 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:36,136 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,136 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:36,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:36,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:36,137 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,137 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:36,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:36,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:36,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:36,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:36,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:36,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:36,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:36,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:36,137 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:36,138 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:36,138 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:36,138 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:36,138 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:36,138 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:36,138 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-24 09:00:36,138 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:36,138 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:36,138 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:36,138 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:36,138 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:36,138 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:36,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:36,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:36,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326836346, max timestamp=1682326836369Num uniques 859; 2023-04-24 09:00:36,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:36,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326836346, max timestamp=1682326836369Num uniques 859; 2023-04-24 09:00:36,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:36,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326836346, max timestamp=1682326836369Num uniques 859; 2023-04-24 09:00:36,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:36,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326896391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326896391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326896394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326896394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,466 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/a04f470c7fbf48788deb0b5f1f82e5e9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a04f470c7fbf48788deb0b5f1f82e5e9 2023-04-24 09:00:36,474 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3695ff097ea6e04d645670eb37341990/B of 3695ff097ea6e04d645670eb37341990 into a04f470c7fbf48788deb0b5f1f82e5e9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:36,474 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:36,474 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/B, priority=12, startTime=1682326836043; duration=0sec 2023-04-24 09:00:36,475 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:36,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326896497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326896498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326896498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326896501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,544 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/f940c20e34224849b491c13a254f9177 2023-04-24 09:00:36,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326896704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326896704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326896705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:36,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326896705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,953 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/494edffde2c240619eab5f050bf938a7 2023-04-24 09:00:36,961 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=731 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/6f7e128b89b74388a91cb2033410f2e7 2023-04-24 09:00:36,964 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/f940c20e34224849b491c13a254f9177 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f940c20e34224849b491c13a254f9177 2023-04-24 09:00:36,967 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f940c20e34224849b491c13a254f9177, entries=150, sequenceid=731, filesize=11.9 K 2023-04-24 09:00:36,967 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/494edffde2c240619eab5f050bf938a7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/494edffde2c240619eab5f050bf938a7 2023-04-24 09:00:36,970 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/494edffde2c240619eab5f050bf938a7, entries=150, sequenceid=731, filesize=11.9 K 2023-04-24 09:00:36,970 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/6f7e128b89b74388a91cb2033410f2e7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6f7e128b89b74388a91cb2033410f2e7 2023-04-24 09:00:36,973 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6f7e128b89b74388a91cb2033410f2e7, entries=150, sequenceid=731, filesize=11.9 K 2023-04-24 09:00:36,973 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 3695ff097ea6e04d645670eb37341990 in 835ms, sequenceid=731, compaction requested=false 2023-04-24 09:00:36,973 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:36,973 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:36,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:36,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:36,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:36,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:36,974 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:37,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:37,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-24 09:00:37,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:37,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:37,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:37,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:37,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:37,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-24 09:00:37,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,022 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,022 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,022 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:37,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36832c36[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:37,023 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:37,023 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326897016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326897017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326897018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:37,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:37,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:37,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,030 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:37,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326897025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,058 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:37,058 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,058 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:37,058 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:37,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:37,060 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:37,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:37,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:37,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:37,060 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:37,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,072 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,073 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:37,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:37,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,081 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:37,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,081 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:37,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:37,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326897126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326897126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326897126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326897137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,181 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:37,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:37,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:37,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:37,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:37,183 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:37,183 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:37,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:37,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,184 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,188 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,188 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,189 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:37,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:37,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:37,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,189 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:37,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:37,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:37,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:37,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:37,190 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:37,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:37,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:37,190 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:37,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,197 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,197 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,197 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:37,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a134ab[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:37,198 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:37,198 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,205 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:37,205 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:37,205 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:37,205 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,206 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:37,213 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:37,213 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,213 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,213 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:37,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:37,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:37,215 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:37,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:37,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:37,215 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:37,215 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:37,222 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,222 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,222 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,222 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,230 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:37,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:37,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,230 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:37,231 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,231 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:37,231 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,331 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:37,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:37,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:37,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:37,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:37,333 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:37,333 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,333 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:37,333 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:37,334 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,334 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326897330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326897330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326897331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,338 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,338 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,339 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:37,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:37,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:37,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:37,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:37,340 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,340 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:37,340 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:37,340 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:37,340 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:37,347 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,347 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:37,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7719ffdb[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:37,348 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:37,348 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326897345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,355 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,355 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:37,355 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:37,355 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:37,355 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:37,363 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:37,363 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,363 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,364 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:37,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:37,365 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:37,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:37,365 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:37,366 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:37,366 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:37,372 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,372 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,372 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,373 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:37,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:37,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,380 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:37,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,380 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:37,381 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:37,381 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,381 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,381 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,381 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/e8c5a113d95a4e0e9ca105439529c5bb 2023-04-24 09:00:37,481 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:37,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:37,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:37,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:37,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:37,483 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:37,483 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,483 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:37,484 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:37,484 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,484 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,488 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,489 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:37,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:37,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:37,490 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,490 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:37,490 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:37,490 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:37,490 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:37,555 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,555 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,555 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,555 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:37,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@425bafb[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:37,557 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:37,557 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,563 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,563 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,564 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:37,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:37,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:37,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:37,572 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:37,572 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,572 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:37,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:37,573 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:37,574 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:37,574 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:37,574 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:37,574 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:37,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,597 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,598 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,611 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:37,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:37,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,612 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:37,612 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:37,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:37,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,611 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,612 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,612 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,612 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,612 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,612 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326897635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326897637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326897637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:37,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326897651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,712 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:37,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:37,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:37,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:37,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:37,714 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:37,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:37,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:37,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,714 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,722 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,722 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,722 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:37,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:37,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:37,723 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,723 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:37,723 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:37,723 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:37,723 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:37,730 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,730 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,730 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,730 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:37,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:37,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b68d4d5[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:37,731 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:37,732 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:37,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:37,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:37,739 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:37,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:37,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:37,749 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:37,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:37,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:37,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:37,749 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:37,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,755 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,764 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,764 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,764 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:37,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:37,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:37,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,764 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,764 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,764 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:37,764 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:37,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/eefb02e551034e6e8103542791dbde1a 2023-04-24 09:00:37,864 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:37,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:37,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:37,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:37,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:37,867 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:37,867 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,867 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:37,867 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:37,867 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,867 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,872 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,872 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:37,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:37,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:37,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:37,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,873 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:37,873 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,873 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:37,873 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:37,873 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:37,873 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:37,880 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,880 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:37,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a0db97f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:37,881 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:37,881 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,888 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,888 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,889 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:37,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:37,889 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:37,889 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:37,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,897 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:37,897 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,897 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:37,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:37,898 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:37,899 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:37,899 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:37,899 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:37,899 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:37,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:37,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,905 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:37,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:37,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:37,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:37,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:37,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:37,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:37,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:37,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:37,914 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:37,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:37,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:37,914 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:37,914 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:38,014 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:38,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:38,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:38,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:38,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:38,017 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:38,017 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:38,017 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:38,017 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:38,017 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,017 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,022 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:38,022 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:38,022 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,022 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:38,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:38,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:38,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:38,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:38,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:38,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:38,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:38,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:38,023 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:38,023 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:38,023 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:38,023 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:38,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:38,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:38,023 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,024 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:38,024 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:38,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,030 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:38,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,030 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:38,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:38,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:38,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725270c2[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:38,032 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:38,032 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,038 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:38,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:38,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,039 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:38,039 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:38,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:38,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:38,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:38,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:38,049 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:38,049 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:38,049 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:38,049 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:38,049 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:38,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:38,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:38,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:38,056 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,057 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:38,081 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:38,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:38,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:38,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:38,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:38,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,081 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,082 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,082 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,082 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,082 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:38,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:38,082 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:38,082 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:38,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326898141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:38,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326898144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326898145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:38,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326898160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,182 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:38,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:38,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:38,185 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:38,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:38,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:38,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:38,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,185 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,197 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,197 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:38,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:38,197 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,197 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:38,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:38,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:38,198 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:38,198 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:38,198 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:38,198 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:38,198 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:38,205 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,205 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:38,205 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,205 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:38,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:38,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b09fe11[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:38,207 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:38,207 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,213 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,213 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,214 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:38,213 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:38,214 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:38,214 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:38,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:38,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:38,222 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,222 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:38,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:38,224 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:38,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:38,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:38,224 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:38,224 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:38,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:38,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,230 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,232 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:38,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:38,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:38,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:38,238 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:38,239 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:38,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:38,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:38,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:38,239 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,239 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:38,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:38,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=770 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/1d722af33d214f2094e62265ef781cc0 2023-04-24 09:00:38,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/e8c5a113d95a4e0e9ca105439529c5bb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e8c5a113d95a4e0e9ca105439529c5bb 2023-04-24 09:00:38,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e8c5a113d95a4e0e9ca105439529c5bb, entries=200, sequenceid=770, filesize=14.3 K 2023-04-24 09:00:38,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/eefb02e551034e6e8103542791dbde1a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/eefb02e551034e6e8103542791dbde1a 2023-04-24 09:00:38,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/eefb02e551034e6e8103542791dbde1a, entries=150, sequenceid=770, filesize=11.9 K 2023-04-24 09:00:38,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/1d722af33d214f2094e62265ef781cc0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d722af33d214f2094e62265ef781cc0 2023-04-24 09:00:38,264 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d722af33d214f2094e62265ef781cc0, entries=150, sequenceid=770, filesize=11.9 K 2023-04-24 09:00:38,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 3695ff097ea6e04d645670eb37341990 in 1253ms, sequenceid=770, compaction requested=true 2023-04-24 09:00:38,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:38,265 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:38,265 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:38,265 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:38,265 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:38,265 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:38,266 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:38,266 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:38,266 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/A is initiating minor compaction (all files) 2023-04-24 09:00:38,266 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/B is initiating minor compaction (all files) 2023-04-24 09:00:38,266 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/A in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:38,266 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/B in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:38,266 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4d086450cef24b1e88403e05191e4804, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f940c20e34224849b491c13a254f9177, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e8c5a113d95a4e0e9ca105439529c5bb] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=38.8 K 2023-04-24 09:00:38,266 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a04f470c7fbf48788deb0b5f1f82e5e9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/494edffde2c240619eab5f050bf938a7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/eefb02e551034e6e8103542791dbde1a] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=36.4 K 2023-04-24 09:00:38,266 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a04f470c7fbf48788deb0b5f1f82e5e9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=720, earliestPutTs=1682326834582 2023-04-24 09:00:38,266 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 4d086450cef24b1e88403e05191e4804, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=720, earliestPutTs=1682326834582 2023-04-24 09:00:38,267 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 494edffde2c240619eab5f050bf938a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1682326835212 2023-04-24 09:00:38,267 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f940c20e34224849b491c13a254f9177, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1682326835212 2023-04-24 09:00:38,267 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting eefb02e551034e6e8103542791dbde1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682326836384 2023-04-24 09:00:38,267 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e8c5a113d95a4e0e9ca105439529c5bb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682326836384 2023-04-24 09:00:38,272 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#B#compaction#412 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:38,272 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#A#compaction#411 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:38,284 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/1cfa7283db42418ea42a8bf2a20d0ebf as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/1cfa7283db42418ea42a8bf2a20d0ebf 2023-04-24 09:00:38,287 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/A of 3695ff097ea6e04d645670eb37341990 into 1cfa7283db42418ea42a8bf2a20d0ebf(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:38,287 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:38,287 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/A, priority=13, startTime=1682326838265; duration=0sec 2023-04-24 09:00:38,287 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:38,287 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:38,288 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:38,288 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/C is initiating minor compaction (all files) 2023-04-24 09:00:38,288 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/C in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:38,288 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6a11eec6ef594280af07e7c0bce3d0d3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6f7e128b89b74388a91cb2033410f2e7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d722af33d214f2094e62265ef781cc0] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=36.4 K 2023-04-24 09:00:38,289 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 6a11eec6ef594280af07e7c0bce3d0d3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=720, earliestPutTs=1682326834582 2023-04-24 09:00:38,289 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 6f7e128b89b74388a91cb2033410f2e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=731, earliestPutTs=1682326835212 2023-04-24 09:00:38,289 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1d722af33d214f2094e62265ef781cc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682326836384 2023-04-24 09:00:38,294 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#C#compaction#413 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:38,302 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/2d48c3b387c04c85bc4556d301972e61 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2d48c3b387c04c85bc4556d301972e61 2023-04-24 09:00:38,306 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/C of 3695ff097ea6e04d645670eb37341990 into 2d48c3b387c04c85bc4556d301972e61(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:38,306 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:38,306 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/C, priority=13, startTime=1682326838265; duration=0sec 2023-04-24 09:00:38,306 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:38,339 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:38,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:38,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:38,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:38,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:38,342 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:38,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:38,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:38,342 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:38,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,353 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:38,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:38,353 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:38,353 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:38,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:38,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:38,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:38,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:38,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:38,354 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:38,354 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:38,354 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:38,355 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:38,355 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:38,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:38,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:38,355 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-24 09:00:38,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:38,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:38,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:38,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:38,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:38,355 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:38,679 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/3d52bc3dd49c4a50884196fcb906cf71 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/3d52bc3dd49c4a50884196fcb906cf71 2023-04-24 09:00:38,682 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/B of 3695ff097ea6e04d645670eb37341990 into 3d52bc3dd49c4a50884196fcb906cf71(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:38,682 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:38,683 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/B, priority=13, startTime=1682326838265; duration=0sec 2023-04-24 09:00:38,683 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:38,768 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/e108238be9b0471397c2d9d9e95681ed 2023-04-24 09:00:38,778 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/4cc47b469fcf4d5298343ebf6640cbd7 2023-04-24 09:00:39,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:39,152 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:39,186 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/0313307f04c94a68aa37ba7ce5759326 2023-04-24 09:00:39,189 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/e108238be9b0471397c2d9d9e95681ed as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e108238be9b0471397c2d9d9e95681ed 2023-04-24 09:00:39,191 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e108238be9b0471397c2d9d9e95681ed, entries=100, sequenceid=780, filesize=9.5 K 2023-04-24 09:00:39,192 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/4cc47b469fcf4d5298343ebf6640cbd7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/4cc47b469fcf4d5298343ebf6640cbd7 2023-04-24 09:00:39,195 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/4cc47b469fcf4d5298343ebf6640cbd7, entries=100, sequenceid=780, filesize=9.5 K 2023-04-24 09:00:39,196 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/0313307f04c94a68aa37ba7ce5759326 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0313307f04c94a68aa37ba7ce5759326 2023-04-24 09:00:39,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326839162, max timestamp=1682326839195Num uniques 859; 2023-04-24 09:00:39,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:39,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326839162, max timestamp=1682326839195Num uniques 859; 2023-04-24 09:00:39,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682326839162, max timestamp=1682326839195Num uniques 859; 2023-04-24 09:00:39,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:39,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-24 09:00:39,201 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0313307f04c94a68aa37ba7ce5759326, entries=100, sequenceid=780, filesize=9.5 K 2023-04-24 09:00:39,201 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=46.96 KB/48090 for 3695ff097ea6e04d645670eb37341990 in 846ms, sequenceid=780, compaction requested=false 2023-04-24 09:00:39,201 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:39,202 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,202 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:39,202 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:39,202 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,202 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:39,202 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:39,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:39,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=53.67 KB heapSize=136.55 KB 2023-04-24 09:00:39,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:39,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:39,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:39,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:39,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:39,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:39,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,238 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,239 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,239 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:39,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36aa47d9[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:39,240 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:39,240 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,247 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,247 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,247 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:39,247 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:39,247 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:39,247 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:39,255 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:39,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,255 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,255 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326899245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326899251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682326839230, max timestamp=1682326839255Num uniques 859; 2023-04-24 09:00:39,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:39,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682326839230, max timestamp=1682326839255Num uniques 859; 2023-04-24 09:00:39,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:39,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682326839230, max timestamp=1682326839255Num uniques 859; 2023-04-24 09:00:39,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:39,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:39,258 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:39,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:39,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:39,258 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:39,258 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:39,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:39,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,272 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,272 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,272 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:39,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326899307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326899308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326899357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326899357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,372 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:39,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:39,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:39,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:39,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:39,375 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:39,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:39,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:39,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:39,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,380 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,380 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,380 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:39,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:39,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:39,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:39,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:39,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:39,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:39,388 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,388 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,389 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:39,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@114bc632[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:39,390 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:39,390 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,397 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,397 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:39,397 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:39,397 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,397 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:39,397 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:39,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:39,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:39,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:39,407 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:39,407 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:39,407 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:39,407 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:39,407 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:39,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,414 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,415 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326899414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326899414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:39,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:39,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:39,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,440 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:39,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,440 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:39,440 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,441 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,441 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,441 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,441 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,541 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:39,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:39,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:39,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:39,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:39,543 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:39,543 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:39,543 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:39,543 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:39,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,544 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,555 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,555 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,555 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,555 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:39,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:39,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:39,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:39,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:39,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:39,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:39,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:39,556 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,556 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:39,556 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:39,556 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,556 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:39,557 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:39,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,563 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:39,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fe70b9[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:39,565 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:39,565 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326899564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326899564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,572 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,572 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:39,572 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,572 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:39,572 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:39,572 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:39,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:39,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,580 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:39,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:39,582 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:39,582 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:39,582 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:39,582 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:39,582 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:39,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,590 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,597 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:39,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,597 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,597 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:39,597 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:39,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326899619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/aef54e7d33b441778ea85564fcb7710c 2023-04-24 09:00:39,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326899619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,697 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:39,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:39,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:39,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:39,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:39,700 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:39,700 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:39,700 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:39,700 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:39,700 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,701 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,705 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,705 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,705 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,705 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:39,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:39,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:39,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:39,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:39,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:39,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:39,706 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:39,706 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,707 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:39,707 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:39,707 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,707 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:39,707 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:39,707 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,707 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:39,707 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:39,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,714 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,714 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:39,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6413f1c1[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:39,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:39,715 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,722 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,722 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:39,722 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,722 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:39,722 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:39,722 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:39,730 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,730 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,730 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,730 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:39,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:39,732 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:39,732 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:39,732 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:39,732 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:39,732 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:39,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:39,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:39,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,747 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,747 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:39,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,847 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:39,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:39,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:39,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:39,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:39,850 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:39,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:39,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:39,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:39,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,850 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326899869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326899870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,916 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:39,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:39,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:39,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:39,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:39,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:39,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:39,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:39,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:39,917 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:39,917 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,917 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. started... 2023-04-24 09:00:39,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. as already flushing 2023-04-24 09:00:39,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:39,918 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:39,918 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:39,918 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,918 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:39,918 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:39,919 DEBUG [Thread-2665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x609213af to 127.0.0.1:63978 2023-04-24 09:00:39,919 DEBUG [Thread-2665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:39,923 DEBUG [Thread-2669] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6acb8caa to 127.0.0.1:63978 2023-04-24 09:00:39,923 DEBUG [Thread-2669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:39,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326899923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,924 DEBUG [Thread-2671] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70263db3 to 127.0.0.1:63978 2023-04-24 09:00:39,924 DEBUG [Thread-2671] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:39,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:39,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326899924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,926 DEBUG [Thread-2667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ee0ad26 to 127.0.0.1:63978 2023-04-24 09:00:39,926 DEBUG [Thread-2667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:39,929 DEBUG [Thread-2673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25d2db45 to 127.0.0.1:63978 2023-04-24 09:00:39,929 DEBUG [Thread-2673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:39,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,939 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,939 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:39,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:39,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e3e615[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:39,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:39,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,947 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:39,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:39,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:39,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:39,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:39,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:39,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:39,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:39,960 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:39,960 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:39,960 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:39,961 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:39,961 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:39,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:39,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:39,973 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,976 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:39,989 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:39,989 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,989 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:39,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:39,989 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:39,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:39,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:39,989 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:39,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:39,989 DEBUG [Thread-2664] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:39,990 DEBUG [Thread-2664] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:39,989 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:39,990 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:39,990 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:39,990 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:39,990 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:40,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/723ca7a7c0ed4059a6df5ad745499ed2 2023-04-24 09:00:40,090 DEBUG [Thread-2664] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:40,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:40,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:40,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57224 deadline: 1682326900377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:40,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:40,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57184 deadline: 1682326900377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:40,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:40,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57162 deadline: 1682326900427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:40,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:40,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:57228 deadline: 1682326900427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:40,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/fabb0a43e64f43738dee7d540c2f888d 2023-04-24 09:00:40,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/aef54e7d33b441778ea85564fcb7710c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/aef54e7d33b441778ea85564fcb7710c 2023-04-24 09:00:40,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/aef54e7d33b441778ea85564fcb7710c, entries=150, sequenceid=805, filesize=11.9 K 2023-04-24 09:00:40,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/723ca7a7c0ed4059a6df5ad745499ed2 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/723ca7a7c0ed4059a6df5ad745499ed2 2023-04-24 09:00:40,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/723ca7a7c0ed4059a6df5ad745499ed2, entries=150, sequenceid=805, filesize=11.9 K 2023-04-24 09:00:40,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/fabb0a43e64f43738dee7d540c2f888d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/fabb0a43e64f43738dee7d540c2f888d 2023-04-24 09:00:40,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/fabb0a43e64f43738dee7d540c2f888d, entries=150, sequenceid=805, filesize=11.9 K 2023-04-24 09:00:40,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~156.42 KB/160176, currentSize=127.47 KB/130530 for 3695ff097ea6e04d645670eb37341990 in 1272ms, sequenceid=805, compaction requested=true 2023-04-24 09:00:40,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:40,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:40,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:40,476 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:40,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:40,476 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:40,477 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:40,477 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:40,477 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/A is initiating minor compaction (all files) 2023-04-24 09:00:40,477 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/B is initiating minor compaction (all files) 2023-04-24 09:00:40,477 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/A in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:40,477 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/B in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:40,477 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/1cfa7283db42418ea42a8bf2a20d0ebf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e108238be9b0471397c2d9d9e95681ed, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/aef54e7d33b441778ea85564fcb7710c] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=34.1 K 2023-04-24 09:00:40,477 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/3d52bc3dd49c4a50884196fcb906cf71, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/4cc47b469fcf4d5298343ebf6640cbd7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/723ca7a7c0ed4059a6df5ad745499ed2] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=34.1 K 2023-04-24 09:00:40,477 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 1cfa7283db42418ea42a8bf2a20d0ebf, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682326836384 2023-04-24 09:00:40,477 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3d52bc3dd49c4a50884196fcb906cf71, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682326836384 2023-04-24 09:00:40,478 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e108238be9b0471397c2d9d9e95681ed, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1682326837014 2023-04-24 09:00:40,478 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4cc47b469fcf4d5298343ebf6640cbd7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1682326837014 2023-04-24 09:00:40,478 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting aef54e7d33b441778ea85564fcb7710c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682326839166 2023-04-24 09:00:40,478 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 723ca7a7c0ed4059a6df5ad745499ed2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682326839166 2023-04-24 09:00:40,486 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#B#compaction#420 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:40,486 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#A#compaction#421 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:40,899 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/ba3aaf89abe44eff8fa166dd3801e49b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ba3aaf89abe44eff8fa166dd3801e49b 2023-04-24 09:00:40,899 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/88851a3bb352443db6991ac19c4d9ecd as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/88851a3bb352443db6991ac19c4d9ecd 2023-04-24 09:00:40,904 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/B of 3695ff097ea6e04d645670eb37341990 into ba3aaf89abe44eff8fa166dd3801e49b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:40,904 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:40,904 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/B, priority=13, startTime=1682326840476; duration=0sec 2023-04-24 09:00:40,904 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/A of 3695ff097ea6e04d645670eb37341990 into 88851a3bb352443db6991ac19c4d9ecd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:40,904 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:40,904 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:40,904 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:40,904 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/A, priority=13, startTime=1682326840476; duration=0sec 2023-04-24 09:00:40,904 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:40,905 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:40,905 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): 3695ff097ea6e04d645670eb37341990/C is initiating minor compaction (all files) 2023-04-24 09:00:40,905 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3695ff097ea6e04d645670eb37341990/C in TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:40,906 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2d48c3b387c04c85bc4556d301972e61, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0313307f04c94a68aa37ba7ce5759326, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/fabb0a43e64f43738dee7d540c2f888d] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp, totalSize=34.1 K 2023-04-24 09:00:40,906 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 2d48c3b387c04c85bc4556d301972e61, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=770, earliestPutTs=1682326836384 2023-04-24 09:00:40,906 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 0313307f04c94a68aa37ba7ce5759326, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1682326837014 2023-04-24 09:00:40,907 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting fabb0a43e64f43738dee7d540c2f888d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682326839166 2023-04-24 09:00:40,913 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): 3695ff097ea6e04d645670eb37341990#C#compaction#422 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:41,333 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/0f687ef6bef14508a91d2e435492a518 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0f687ef6bef14508a91d2e435492a518 2023-04-24 09:00:41,340 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3695ff097ea6e04d645670eb37341990/C of 3695ff097ea6e04d645670eb37341990 into 0f687ef6bef14508a91d2e435492a518(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:41,340 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:41,340 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990., storeName=3695ff097ea6e04d645670eb37341990/C, priority=13, startTime=1682326840476; duration=0sec 2023-04-24 09:00:41,340 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:41,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:41,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-24 09:00:41,384 DEBUG [Thread-2660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62447900 to 127.0.0.1:63978 2023-04-24 09:00:41,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:41,384 DEBUG [Thread-2660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:41,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:41,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:41,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:41,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:41,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:41,389 DEBUG [Thread-2654] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0817e2e5 to 127.0.0.1:63978 2023-04-24 09:00:41,389 DEBUG [Thread-2654] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:41,430 DEBUG [Thread-2656] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4048dcb4 to 127.0.0.1:63978 2023-04-24 09:00:41,430 DEBUG [Thread-2656] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:41,435 DEBUG [Thread-2658] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e7a5e9a to 127.0.0.1:63978 2023-04-24 09:00:41,436 DEBUG [Thread-2658] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:41,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/0a0f4d94269743aab5a6308232bc653b 2023-04-24 09:00:42,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/7ee76f0b22bb464bab8fb6052539fa8d 2023-04-24 09:00:42,217 DEBUG [Thread-2662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x519db5f2 to 127.0.0.1:63978 2023-04-24 09:00:42,217 DEBUG [Thread-2662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:42,217 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 09:00:42,218 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 194 2023-04-24 09:00:42,218 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 138 2023-04-24 09:00:42,218 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 188 2023-04-24 09:00:42,218 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 215 2023-04-24 09:00:42,218 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-24 09:00:42,219 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 09:00:42,219 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 09:00:42,219 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2348 2023-04-24 09:00:42,219 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7038 rows 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2348 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7043 rows 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2354 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7055 rows 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2336 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7001 rows 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(402): scanned 2350 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(403): verified 7046 rows 2023-04-24 09:00:42,220 INFO [Listener at localhost.localdomain/44291] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 09:00:42,220 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4df55fe5 to 127.0.0.1:63978 2023-04-24 09:00:42,221 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:00:42,222 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 09:00:42,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-24 09:00:42,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:42,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 09:00:42,229 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326842228"}]},"ts":"1682326842228"} 2023-04-24 09:00:42,229 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 09:00:42,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/81e126a24a6e4ebe9c79b1a59a233a54 2023-04-24 09:00:42,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/0a0f4d94269743aab5a6308232bc653b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/0a0f4d94269743aab5a6308232bc653b 2023-04-24 09:00:42,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/0a0f4d94269743aab5a6308232bc653b, entries=150, sequenceid=844, filesize=11.9 K 2023-04-24 09:00:42,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/7ee76f0b22bb464bab8fb6052539fa8d as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7ee76f0b22bb464bab8fb6052539fa8d 2023-04-24 09:00:42,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7ee76f0b22bb464bab8fb6052539fa8d, entries=150, sequenceid=844, filesize=11.9 K 2023-04-24 09:00:42,240 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/81e126a24a6e4ebe9c79b1a59a233a54 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/81e126a24a6e4ebe9c79b1a59a233a54 2023-04-24 09:00:42,243 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/81e126a24a6e4ebe9c79b1a59a233a54, entries=150, sequenceid=844, filesize=11.9 K 2023-04-24 09:00:42,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=26.84 KB/27480 for 3695ff097ea6e04d645670eb37341990 in 860ms, sequenceid=844, compaction requested=false 2023-04-24 09:00:42,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:42,263 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 09:00:42,264 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3695ff097ea6e04d645670eb37341990, UNASSIGN}] 2023-04-24 09:00:42,265 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3695ff097ea6e04d645670eb37341990, UNASSIGN 2023-04-24 09:00:42,265 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=3695ff097ea6e04d645670eb37341990, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:42,266 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326842265"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326842265"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326842265"}]},"ts":"1682326842265"} 2023-04-24 09:00:42,267 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 09:00:42,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 09:00:42,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:42,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 3695ff097ea6e04d645670eb37341990, disabling compactions & flushes 2023-04-24 09:00:42,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:42,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:42,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. after waiting 0 ms 2023-04-24 09:00:42,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:42,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 3695ff097ea6e04d645670eb37341990 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 09:00:42,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=A 2023-04-24 09:00:42,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:42,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=B 2023-04-24 09:00:42,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:42,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3695ff097ea6e04d645670eb37341990, store=C 2023-04-24 09:00:42,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:42,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 09:00:42,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 09:00:42,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=851 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/06cd5fc57707482e8f4e090c1c805660 2023-04-24 09:00:42,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=851 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/60a5ad71964946de9ab7401394100ea3 2023-04-24 09:00:43,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=851 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/e158f03549e344b6a317bf06e3543e92 2023-04-24 09:00:43,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/A/06cd5fc57707482e8f4e090c1c805660 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/06cd5fc57707482e8f4e090c1c805660 2023-04-24 09:00:43,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/06cd5fc57707482e8f4e090c1c805660, entries=150, sequenceid=851, filesize=11.9 K 2023-04-24 09:00:43,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/B/60a5ad71964946de9ab7401394100ea3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/60a5ad71964946de9ab7401394100ea3 2023-04-24 09:00:43,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/60a5ad71964946de9ab7401394100ea3, entries=150, sequenceid=851, filesize=11.9 K 2023-04-24 09:00:43,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/.tmp/C/e158f03549e344b6a317bf06e3543e92 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e158f03549e344b6a317bf06e3543e92 2023-04-24 09:00:43,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e158f03549e344b6a317bf06e3543e92, entries=150, sequenceid=851, filesize=11.9 K 2023-04-24 09:00:43,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 3695ff097ea6e04d645670eb37341990 in 869ms, sequenceid=851, compaction requested=true 2023-04-24 09:00:43,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4ea9df232bc34eee903558f7809a4aa3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/6b67cc97903547339a6bac19c0e2c94d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/eb4e0f4a1c234db397e623a904debcc9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/454a1277af684f949d968a1f8ea301e5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/368291fdc5c84be18d214db6c12135ca, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/814e0716e1fa494a99b3710f47c60c14, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e86d1345e5f6482fa7e3016b6ccb7633, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/c30b4ce9e5e84d6ab967d458db11dd2e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e27287df98ad4c2b923f93854106f6d1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/5bedd0acda4f4d8a9fdd95e8216f21f8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9cbc0f2979b142f890b114f902ed79bb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f576161bd0524272b57e3798de835dda, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/be9c81b31bb9491faeb4e0bf68ed91f1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/36cd4b3d75d74732af9b1a12e33ccc49, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/2c33f8bf64a147259d4b3133f301c289, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/154c271f057b4663982bd6f9cc786222, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/76788337b2e8484cb4f4eaf626b173d6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/891a67b7d6694ab9981c30e9db651970, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/beea66d1f4cd49b6b4a12408efcef73d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/d13c903a32f5474ab9afd92a35733c1a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9f62d061fba2476d9f0e6fd3cbcde1bb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/03c1dba1ff464b3cb4f85db27a1ae70f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4d086450cef24b1e88403e05191e4804, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f940c20e34224849b491c13a254f9177, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e8c5a113d95a4e0e9ca105439529c5bb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/1cfa7283db42418ea42a8bf2a20d0ebf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e108238be9b0471397c2d9d9e95681ed, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/aef54e7d33b441778ea85564fcb7710c] to archive 2023-04-24 09:00:43,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:00:43,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4ea9df232bc34eee903558f7809a4aa3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4ea9df232bc34eee903558f7809a4aa3 2023-04-24 09:00:43,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/6b67cc97903547339a6bac19c0e2c94d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/6b67cc97903547339a6bac19c0e2c94d 2023-04-24 09:00:43,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/eb4e0f4a1c234db397e623a904debcc9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/eb4e0f4a1c234db397e623a904debcc9 2023-04-24 09:00:43,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/454a1277af684f949d968a1f8ea301e5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/454a1277af684f949d968a1f8ea301e5 2023-04-24 09:00:43,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/368291fdc5c84be18d214db6c12135ca to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/368291fdc5c84be18d214db6c12135ca 2023-04-24 09:00:43,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/814e0716e1fa494a99b3710f47c60c14 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/814e0716e1fa494a99b3710f47c60c14 2023-04-24 09:00:43,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e86d1345e5f6482fa7e3016b6ccb7633 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e86d1345e5f6482fa7e3016b6ccb7633 2023-04-24 09:00:43,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/c30b4ce9e5e84d6ab967d458db11dd2e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/c30b4ce9e5e84d6ab967d458db11dd2e 2023-04-24 09:00:43,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e27287df98ad4c2b923f93854106f6d1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e27287df98ad4c2b923f93854106f6d1 2023-04-24 09:00:43,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/5bedd0acda4f4d8a9fdd95e8216f21f8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/5bedd0acda4f4d8a9fdd95e8216f21f8 2023-04-24 09:00:43,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9cbc0f2979b142f890b114f902ed79bb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9cbc0f2979b142f890b114f902ed79bb 2023-04-24 09:00:43,320 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f576161bd0524272b57e3798de835dda to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f576161bd0524272b57e3798de835dda 2023-04-24 09:00:43,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/be9c81b31bb9491faeb4e0bf68ed91f1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/be9c81b31bb9491faeb4e0bf68ed91f1 2023-04-24 09:00:43,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/36cd4b3d75d74732af9b1a12e33ccc49 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/36cd4b3d75d74732af9b1a12e33ccc49 2023-04-24 09:00:43,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/2c33f8bf64a147259d4b3133f301c289 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/2c33f8bf64a147259d4b3133f301c289 2023-04-24 09:00:43,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/154c271f057b4663982bd6f9cc786222 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/154c271f057b4663982bd6f9cc786222 2023-04-24 09:00:43,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/76788337b2e8484cb4f4eaf626b173d6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/76788337b2e8484cb4f4eaf626b173d6 2023-04-24 09:00:43,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/891a67b7d6694ab9981c30e9db651970 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/891a67b7d6694ab9981c30e9db651970 2023-04-24 09:00:43,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/beea66d1f4cd49b6b4a12408efcef73d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/beea66d1f4cd49b6b4a12408efcef73d 2023-04-24 09:00:43,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/d13c903a32f5474ab9afd92a35733c1a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/d13c903a32f5474ab9afd92a35733c1a 2023-04-24 09:00:43,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9f62d061fba2476d9f0e6fd3cbcde1bb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/9f62d061fba2476d9f0e6fd3cbcde1bb 2023-04-24 09:00:43,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/03c1dba1ff464b3cb4f85db27a1ae70f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/03c1dba1ff464b3cb4f85db27a1ae70f 2023-04-24 09:00:43,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4d086450cef24b1e88403e05191e4804 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/4d086450cef24b1e88403e05191e4804 2023-04-24 09:00:43,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f940c20e34224849b491c13a254f9177 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/f940c20e34224849b491c13a254f9177 2023-04-24 09:00:43,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e8c5a113d95a4e0e9ca105439529c5bb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e8c5a113d95a4e0e9ca105439529c5bb 2023-04-24 09:00:43,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/1cfa7283db42418ea42a8bf2a20d0ebf to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/1cfa7283db42418ea42a8bf2a20d0ebf 2023-04-24 09:00:43,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e108238be9b0471397c2d9d9e95681ed to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/e108238be9b0471397c2d9d9e95681ed 2023-04-24 09:00:43,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/aef54e7d33b441778ea85564fcb7710c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/aef54e7d33b441778ea85564fcb7710c 2023-04-24 09:00:43,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 09:00:43,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/5ebf45994f3044eb8c91eb2968522be4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/42dcfe49f706446599f1372caa90d712, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/285f2fb94edd4a989499191c2901acaf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/b7bd1cdcaea14439a1676e5f0fb05715, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/56b7f49bbee84b16b826dccb69456a1d, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f737a999301c4bb2b552763f489c3bd3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7a8911391d5e45a2923d95b89a1766ac, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ee2b74f0edb34f98bb371f8884d2dabd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/2ecda413122b4489a6e17f518b7feda5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/cf3a79678eb74f88b3ff72b44c893655, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a769f546d0e34a5db56ae66702fccfb7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/779a9733c25d4bdbb50dfc9c52f5ca37, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/8147be04f4bc4bde8750c829ebe1f691, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f9cf09d8bfe449779b507f27b1c95493, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/c83f14ff3d2a4dcfa5470c6e99b2d6d6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/764cb295cf544258976fed54594cf116, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ec45c2a95b3e4579b1716604a5394ca5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/737ebc246a824788a5a6f9d9bef705cc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f14e273f886641c4880971cbb7edd984, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/273880aeb5954ba8bf01efc851bd58fb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/6a5e842addf44a6f99fbbf6298eb5827, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a04f470c7fbf48788deb0b5f1f82e5e9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/d3d4ef2bd4ce4464bc1d8f93fb93ca03, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/494edffde2c240619eab5f050bf938a7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/3d52bc3dd49c4a50884196fcb906cf71, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/eefb02e551034e6e8103542791dbde1a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/4cc47b469fcf4d5298343ebf6640cbd7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/723ca7a7c0ed4059a6df5ad745499ed2] to archive 2023-04-24 09:00:43,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:00:43,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/5ebf45994f3044eb8c91eb2968522be4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/5ebf45994f3044eb8c91eb2968522be4 2023-04-24 09:00:43,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/42dcfe49f706446599f1372caa90d712 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/42dcfe49f706446599f1372caa90d712 2023-04-24 09:00:43,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/285f2fb94edd4a989499191c2901acaf to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/285f2fb94edd4a989499191c2901acaf 2023-04-24 09:00:43,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/b7bd1cdcaea14439a1676e5f0fb05715 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/b7bd1cdcaea14439a1676e5f0fb05715 2023-04-24 09:00:43,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/56b7f49bbee84b16b826dccb69456a1d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/56b7f49bbee84b16b826dccb69456a1d 2023-04-24 09:00:43,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f737a999301c4bb2b552763f489c3bd3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f737a999301c4bb2b552763f489c3bd3 2023-04-24 09:00:43,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7a8911391d5e45a2923d95b89a1766ac to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7a8911391d5e45a2923d95b89a1766ac 2023-04-24 09:00:43,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ee2b74f0edb34f98bb371f8884d2dabd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ee2b74f0edb34f98bb371f8884d2dabd 2023-04-24 09:00:43,342 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/2ecda413122b4489a6e17f518b7feda5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/2ecda413122b4489a6e17f518b7feda5 2023-04-24 09:00:43,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/cf3a79678eb74f88b3ff72b44c893655 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/cf3a79678eb74f88b3ff72b44c893655 2023-04-24 09:00:43,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a769f546d0e34a5db56ae66702fccfb7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a769f546d0e34a5db56ae66702fccfb7 2023-04-24 09:00:43,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/779a9733c25d4bdbb50dfc9c52f5ca37 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/779a9733c25d4bdbb50dfc9c52f5ca37 2023-04-24 09:00:43,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/8147be04f4bc4bde8750c829ebe1f691 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/8147be04f4bc4bde8750c829ebe1f691 2023-04-24 09:00:43,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f9cf09d8bfe449779b507f27b1c95493 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f9cf09d8bfe449779b507f27b1c95493 2023-04-24 09:00:43,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/c83f14ff3d2a4dcfa5470c6e99b2d6d6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/c83f14ff3d2a4dcfa5470c6e99b2d6d6 2023-04-24 09:00:43,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/764cb295cf544258976fed54594cf116 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/764cb295cf544258976fed54594cf116 2023-04-24 09:00:43,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ec45c2a95b3e4579b1716604a5394ca5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ec45c2a95b3e4579b1716604a5394ca5 2023-04-24 09:00:43,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/737ebc246a824788a5a6f9d9bef705cc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/737ebc246a824788a5a6f9d9bef705cc 2023-04-24 09:00:43,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f14e273f886641c4880971cbb7edd984 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/f14e273f886641c4880971cbb7edd984 2023-04-24 09:00:43,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/273880aeb5954ba8bf01efc851bd58fb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/273880aeb5954ba8bf01efc851bd58fb 2023-04-24 09:00:43,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/6a5e842addf44a6f99fbbf6298eb5827 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/6a5e842addf44a6f99fbbf6298eb5827 2023-04-24 09:00:43,351 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a04f470c7fbf48788deb0b5f1f82e5e9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/a04f470c7fbf48788deb0b5f1f82e5e9 2023-04-24 09:00:43,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/d3d4ef2bd4ce4464bc1d8f93fb93ca03 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/d3d4ef2bd4ce4464bc1d8f93fb93ca03 2023-04-24 09:00:43,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/494edffde2c240619eab5f050bf938a7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/494edffde2c240619eab5f050bf938a7 2023-04-24 09:00:43,353 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/3d52bc3dd49c4a50884196fcb906cf71 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/3d52bc3dd49c4a50884196fcb906cf71 2023-04-24 09:00:43,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/eefb02e551034e6e8103542791dbde1a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/eefb02e551034e6e8103542791dbde1a 2023-04-24 09:00:43,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/4cc47b469fcf4d5298343ebf6640cbd7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/4cc47b469fcf4d5298343ebf6640cbd7 2023-04-24 09:00:43,355 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/723ca7a7c0ed4059a6df5ad745499ed2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/723ca7a7c0ed4059a6df5ad745499ed2 2023-04-24 09:00:43,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/715ae55b03b047fda5816e8f075b05b6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/7097c984a8664405bfcf3b0a9c498669, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/94f03489dfe740ad92ba7974aab22cce, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/398cdc841a974933a2525f23da2cb610, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e2ed58a69f884ae986312c7fc1009e64, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/9e41673e73c7423ab5d1e4428985467b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/3e90baa41d2540c3980e542c1e8351bd, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/718dfa3e18144d75b878861bc26a84e1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e1b5b4f241604d1b89fcf3b9cf18b26e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dbad1b82bf684c59a045a97d87653092, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/676b2e94075140ed98baef23d6f53b8f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/a36f91855efd4e41b35b597f16d069d3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dd87ef75633742089aa2844d24b9354a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/d6511b121b204add88b848140b3617a5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/febc176f1fd64f2b8f43af638eadde81, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/212e120037d14a8b9cefd8dd1f1c2198, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d33ac8b663b433e93d56f8ad25e3891, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2b2c7099425549ef9ef28a4349690136, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/15f4b11bf3164b6d8db3815e4c20c841, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/4bd256099ab2429fa06ce7f9166538e6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6d79a7920d3b4a4e872a85feb704eb0e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6a11eec6ef594280af07e7c0bce3d0d3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/233b1ac3d35442749a7901a609a847d5, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6f7e128b89b74388a91cb2033410f2e7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2d48c3b387c04c85bc4556d301972e61, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d722af33d214f2094e62265ef781cc0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0313307f04c94a68aa37ba7ce5759326, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/fabb0a43e64f43738dee7d540c2f888d] to archive 2023-04-24 09:00:43,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:00:43,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/715ae55b03b047fda5816e8f075b05b6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/715ae55b03b047fda5816e8f075b05b6 2023-04-24 09:00:43,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/7097c984a8664405bfcf3b0a9c498669 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/7097c984a8664405bfcf3b0a9c498669 2023-04-24 09:00:43,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/94f03489dfe740ad92ba7974aab22cce to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/94f03489dfe740ad92ba7974aab22cce 2023-04-24 09:00:43,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/398cdc841a974933a2525f23da2cb610 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/398cdc841a974933a2525f23da2cb610 2023-04-24 09:00:43,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e2ed58a69f884ae986312c7fc1009e64 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e2ed58a69f884ae986312c7fc1009e64 2023-04-24 09:00:43,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/9e41673e73c7423ab5d1e4428985467b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/9e41673e73c7423ab5d1e4428985467b 2023-04-24 09:00:43,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/3e90baa41d2540c3980e542c1e8351bd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/3e90baa41d2540c3980e542c1e8351bd 2023-04-24 09:00:43,364 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/718dfa3e18144d75b878861bc26a84e1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/718dfa3e18144d75b878861bc26a84e1 2023-04-24 09:00:43,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e1b5b4f241604d1b89fcf3b9cf18b26e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e1b5b4f241604d1b89fcf3b9cf18b26e 2023-04-24 09:00:43,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dbad1b82bf684c59a045a97d87653092 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dbad1b82bf684c59a045a97d87653092 2023-04-24 09:00:43,367 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/676b2e94075140ed98baef23d6f53b8f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/676b2e94075140ed98baef23d6f53b8f 2023-04-24 09:00:43,368 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/a36f91855efd4e41b35b597f16d069d3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/a36f91855efd4e41b35b597f16d069d3 2023-04-24 09:00:43,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dd87ef75633742089aa2844d24b9354a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/dd87ef75633742089aa2844d24b9354a 2023-04-24 09:00:43,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/d6511b121b204add88b848140b3617a5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/d6511b121b204add88b848140b3617a5 2023-04-24 09:00:43,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/febc176f1fd64f2b8f43af638eadde81 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/febc176f1fd64f2b8f43af638eadde81 2023-04-24 09:00:43,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/212e120037d14a8b9cefd8dd1f1c2198 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/212e120037d14a8b9cefd8dd1f1c2198 2023-04-24 09:00:43,372 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d33ac8b663b433e93d56f8ad25e3891 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d33ac8b663b433e93d56f8ad25e3891 2023-04-24 09:00:43,373 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2b2c7099425549ef9ef28a4349690136 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2b2c7099425549ef9ef28a4349690136 2023-04-24 09:00:43,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/15f4b11bf3164b6d8db3815e4c20c841 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/15f4b11bf3164b6d8db3815e4c20c841 2023-04-24 09:00:43,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/4bd256099ab2429fa06ce7f9166538e6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/4bd256099ab2429fa06ce7f9166538e6 2023-04-24 09:00:43,376 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6d79a7920d3b4a4e872a85feb704eb0e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6d79a7920d3b4a4e872a85feb704eb0e 2023-04-24 09:00:43,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6a11eec6ef594280af07e7c0bce3d0d3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6a11eec6ef594280af07e7c0bce3d0d3 2023-04-24 09:00:43,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/233b1ac3d35442749a7901a609a847d5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/233b1ac3d35442749a7901a609a847d5 2023-04-24 09:00:43,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6f7e128b89b74388a91cb2033410f2e7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/6f7e128b89b74388a91cb2033410f2e7 2023-04-24 09:00:43,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2d48c3b387c04c85bc4556d301972e61 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/2d48c3b387c04c85bc4556d301972e61 2023-04-24 09:00:43,381 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d722af33d214f2094e62265ef781cc0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/1d722af33d214f2094e62265ef781cc0 2023-04-24 09:00:43,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0313307f04c94a68aa37ba7ce5759326 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0313307f04c94a68aa37ba7ce5759326 2023-04-24 09:00:43,383 DEBUG [StoreCloser-TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/fabb0a43e64f43738dee7d540c2f888d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/fabb0a43e64f43738dee7d540c2f888d 2023-04-24 09:00:43,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/recovered.edits/854.seqid, newMaxSeqId=854, maxSeqId=1 2023-04-24 09:00:43,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990. 2023-04-24 09:00:43,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 3695ff097ea6e04d645670eb37341990: 2023-04-24 09:00:43,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed 3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:43,393 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=3695ff097ea6e04d645670eb37341990, regionState=CLOSED 2023-04-24 09:00:43,393 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326843393"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326843393"}]},"ts":"1682326843393"} 2023-04-24 09:00:43,396 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-24 09:00:43,396 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 3695ff097ea6e04d645670eb37341990, server=jenkins-hbase16.apache.org,42837,1682326693550 in 1.1270 sec 2023-04-24 09:00:43,398 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-24 09:00:43,398 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3695ff097ea6e04d645670eb37341990, UNASSIGN in 1.1320 sec 2023-04-24 09:00:43,399 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326843399"}]},"ts":"1682326843399"} 2023-04-24 09:00:43,400 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 09:00:43,414 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 09:00:43,416 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1920 sec 2023-04-24 09:00:44,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-24 09:00:44,336 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-24 09:00:44,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-24 09:00:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:44,343 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:44,344 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:44,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-24 09:00:44,349 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:44,352 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/recovered.edits] 2023-04-24 09:00:44,357 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/06cd5fc57707482e8f4e090c1c805660 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/06cd5fc57707482e8f4e090c1c805660 2023-04-24 09:00:44,359 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/0a0f4d94269743aab5a6308232bc653b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/0a0f4d94269743aab5a6308232bc653b 2023-04-24 09:00:44,361 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/88851a3bb352443db6991ac19c4d9ecd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/A/88851a3bb352443db6991ac19c4d9ecd 2023-04-24 09:00:44,365 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/60a5ad71964946de9ab7401394100ea3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/60a5ad71964946de9ab7401394100ea3 2023-04-24 09:00:44,367 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7ee76f0b22bb464bab8fb6052539fa8d to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/7ee76f0b22bb464bab8fb6052539fa8d 2023-04-24 09:00:44,369 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ba3aaf89abe44eff8fa166dd3801e49b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/B/ba3aaf89abe44eff8fa166dd3801e49b 2023-04-24 09:00:44,373 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0f687ef6bef14508a91d2e435492a518 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/0f687ef6bef14508a91d2e435492a518 2023-04-24 09:00:44,375 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/81e126a24a6e4ebe9c79b1a59a233a54 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/81e126a24a6e4ebe9c79b1a59a233a54 2023-04-24 09:00:44,377 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e158f03549e344b6a317bf06e3543e92 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/C/e158f03549e344b6a317bf06e3543e92 2023-04-24 09:00:44,381 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/recovered.edits/854.seqid to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990/recovered.edits/854.seqid 2023-04-24 09:00:44,381 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/3695ff097ea6e04d645670eb37341990 2023-04-24 09:00:44,381 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 09:00:44,384 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:44,386 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 09:00:44,388 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 09:00:44,389 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:44,389 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 09:00:44,389 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682326844389"}]},"ts":"9223372036854775807"} 2023-04-24 09:00:44,391 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 09:00:44,391 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3695ff097ea6e04d645670eb37341990, NAME => 'TestAcidGuarantees,,1682326818618.3695ff097ea6e04d645670eb37341990.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 09:00:44,391 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 09:00:44,392 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682326844392"}]},"ts":"9223372036854775807"} 2023-04-24 09:00:44,394 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 09:00:44,432 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:44,435 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 94 msec 2023-04-24 09:00:44,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-24 09:00:44,447 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-24 09:00:44,463 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230 (was 229) - Thread LEAK? -, OpenFileDescriptor=547 (was 544) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=339 (was 305) - SystemLoadAverage LEAK? -, ProcessCount=165 (was 165), AvailableMemoryMB=8389 (was 8318) - AvailableMemoryMB LEAK? - 2023-04-24 09:00:44,476 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=230, OpenFileDescriptor=547, MaxFileDescriptor=60000, SystemLoadAverage=339, ProcessCount=165, AvailableMemoryMB=8388 2023-04-24 09:00:44,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 09:00:44,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 09:00:44,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:44,481 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-24 09:00:44,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(697): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-24 09:00:44,482 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-24 09:00:44,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 09:00:44,483 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:44,484 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da empty. 2023-04-24 09:00:44,484 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:44,484 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 09:00:44,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 09:00:44,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 09:00:44,893 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 09:00:44,895 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d6e3ad0fadc5b98d862d4692cd3586da, NAME => 'TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp 2023-04-24 09:00:44,901 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 09:00:44,901 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d6e3ad0fadc5b98d862d4692cd3586da, disabling compactions & flushes 2023-04-24 09:00:44,901 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:44,901 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:44,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. after waiting 0 ms 2023-04-24 09:00:44,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:44,902 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:44,902 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:44,904 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-24 09:00:44,905 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326844905"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326844905"}]},"ts":"1682326844905"} 2023-04-24 09:00:44,906 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-24 09:00:44,906 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-24 09:00:44,906 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326844906"}]},"ts":"1682326844906"} 2023-04-24 09:00:44,907 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-24 09:00:44,947 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, ASSIGN}] 2023-04-24 09:00:44,949 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, ASSIGN 2023-04-24 09:00:44,949 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=false 2023-04-24 09:00:45,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 09:00:45,100 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d6e3ad0fadc5b98d862d4692cd3586da, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:45,101 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326845100"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326845100"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326845100"}]},"ts":"1682326845100"} 2023-04-24 09:00:45,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 09:00:45,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:45,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d6e3ad0fadc5b98d862d4692cd3586da, NAME => 'TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.', STARTKEY => '', ENDKEY => ''} 2023-04-24 09:00:45,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 09:00:45,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,264 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,266 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A 2023-04-24 09:00:45,266 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A 2023-04-24 09:00:45,267 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:45,267 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e3ad0fadc5b98d862d4692cd3586da columnFamilyName A 2023-04-24 09:00:45,268 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(310): Store=d6e3ad0fadc5b98d862d4692cd3586da/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:45,268 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,270 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B 2023-04-24 09:00:45,270 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B 2023-04-24 09:00:45,270 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:45,271 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e3ad0fadc5b98d862d4692cd3586da columnFamilyName B 2023-04-24 09:00:45,271 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(310): Store=d6e3ad0fadc5b98d862d4692cd3586da/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:45,272 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,274 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C 2023-04-24 09:00:45,274 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C 2023-04-24 09:00:45,274 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:45,274 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e3ad0fadc5b98d862d4692cd3586da columnFamilyName C 2023-04-24 09:00:45,275 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(310): Store=d6e3ad0fadc5b98d862d4692cd3586da/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:45,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:45,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 09:00:45,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:45,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-24 09:00:45,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened d6e3ad0fadc5b98d862d4692cd3586da; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70849681, jitterRate=0.05574251711368561}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 09:00:45,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:45,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., pid=56, masterSystemTime=1682326845253 2023-04-24 09:00:45,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:45,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:45,289 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d6e3ad0fadc5b98d862d4692cd3586da, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:45,289 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326845288"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326845288"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326845288"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326845288"}]},"ts":"1682326845288"} 2023-04-24 09:00:45,292 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-24 09:00:45,292 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 in 188 msec 2023-04-24 09:00:45,294 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-24 09:00:45,294 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, ASSIGN in 345 msec 2023-04-24 09:00:45,295 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-24 09:00:45,295 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326845295"}]},"ts":"1682326845295"} 2023-04-24 09:00:45,296 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-24 09:00:45,306 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-24 09:00:45,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 828 msec 2023-04-24 09:00:45,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-24 09:00:45,591 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-24 09:00:45,592 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x085873d0 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:45,632 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@754289a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:45,635 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:45,638 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34838, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:45,640 DEBUG [Listener at localhost.localdomain/44291] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-24 09:00:45,643 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:43816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-24 09:00:45,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-24 09:00:45,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$12(2454): Client=jenkins//188.40.62.62 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-24 09:00:45,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-24 09:00:46,055 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 09:00:46,056 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-24 09:00:46,056 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-24 09:00:46,058 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-24 09:00:46,061 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, REOPEN/MOVE}] 2023-04-24 09:00:46,061 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, REOPEN/MOVE 2023-04-24 09:00:46,062 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d6e3ad0fadc5b98d862d4692cd3586da, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,062 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326846062"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326846062"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326846062"}]},"ts":"1682326846062"} 2023-04-24 09:00:46,063 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 09:00:46,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing d6e3ad0fadc5b98d862d4692cd3586da, disabling compactions & flushes 2023-04-24 09:00:46,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. after waiting 0 ms 2023-04-24 09:00:46,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-24 09:00:46,228 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,228 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:46,228 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(3509): Not adding moved region record: d6e3ad0fadc5b98d862d4692cd3586da to self. 2023-04-24 09:00:46,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,230 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d6e3ad0fadc5b98d862d4692cd3586da, regionState=CLOSED 2023-04-24 09:00:46,230 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326846230"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326846230"}]},"ts":"1682326846230"} 2023-04-24 09:00:46,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-24 09:00:46,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 in 168 msec 2023-04-24 09:00:46,233 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase16.apache.org,42837,1682326693550; forceNewPlan=false, retain=true 2023-04-24 09:00:46,384 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d6e3ad0fadc5b98d862d4692cd3586da, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,384 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326846384"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326846384"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326846384"}]},"ts":"1682326846384"} 2023-04-24 09:00:46,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 09:00:46,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d6e3ad0fadc5b98d862d4692cd3586da, NAME => 'TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.', STARTKEY => '', ENDKEY => ''} 2023-04-24 09:00:46,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-24 09:00:46,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,547 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,548 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A 2023-04-24 09:00:46,549 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A 2023-04-24 09:00:46,549 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:46,549 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e3ad0fadc5b98d862d4692cd3586da columnFamilyName A 2023-04-24 09:00:46,552 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(310): Store=d6e3ad0fadc5b98d862d4692cd3586da/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:46,552 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,553 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B 2023-04-24 09:00:46,553 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B 2023-04-24 09:00:46,554 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:46,554 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e3ad0fadc5b98d862d4692cd3586da columnFamilyName B 2023-04-24 09:00:46,554 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(310): Store=d6e3ad0fadc5b98d862d4692cd3586da/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:46,555 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,555 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C 2023-04-24 09:00:46,556 DEBUG [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C 2023-04-24 09:00:46,556 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-24 09:00:46,556 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6e3ad0fadc5b98d862d4692cd3586da columnFamilyName C 2023-04-24 09:00:46,556 INFO [StoreOpener-d6e3ad0fadc5b98d862d4692cd3586da-1] regionserver.HStore(310): Store=d6e3ad0fadc5b98d862d4692cd3586da/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-24 09:00:46,557 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-24 09:00:46,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened d6e3ad0fadc5b98d862d4692cd3586da; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71282114, jitterRate=0.06218627095222473}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-24 09:00:46,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:46,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., pid=61, masterSystemTime=1682326846538 2023-04-24 09:00:46,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,563 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d6e3ad0fadc5b98d862d4692cd3586da, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,563 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326846563"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1682326846563"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682326846563"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682326846563"}]},"ts":"1682326846563"} 2023-04-24 09:00:46,565 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-24 09:00:46,565 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 in 178 msec 2023-04-24 09:00:46,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-24 09:00:46,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, REOPEN/MOVE in 504 msec 2023-04-24 09:00:46,567 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-24 09:00:46,567 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 508 msec 2023-04-24 09:00:46,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 922 msec 2023-04-24 09:00:46,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-24 09:00:46,569 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x768816e9 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,606 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d853d1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,606 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x27acfde4 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,621 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@112588a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,621 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x00fcbd84 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,637 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@407f33a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,638 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x18884189 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,654 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7dfddf81, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,655 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x2ef34de3 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,671 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74adc5bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,671 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x27489815 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,687 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d52316c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,687 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x33765ee2 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,698 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@509de0a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,698 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x16e1a968 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,714 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17e16418, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,715 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x17856520 to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,729 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c342451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,730 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(139): Connect 0x0a82cded to 127.0.0.1:63978 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-24 09:00:46,746 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@361215f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-24 09:00:46,757 DEBUG [hconnection-0x3a22463f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,758 DEBUG [hconnection-0x65b47b4b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,758 DEBUG [hconnection-0x2b626de4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,758 DEBUG [hconnection-0x4e267027-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,758 DEBUG [hconnection-0x6dc74868-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,758 DEBUG [hconnection-0x48b2d7a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,759 DEBUG [hconnection-0x24c20c0e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,761 DEBUG [hconnection-0x1d3da326-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,761 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34864, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,761 DEBUG [hconnection-0x343b9199-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,761 DEBUG [hconnection-0x22c6a44d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-24 09:00:46,762 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34848, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,762 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34858, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,763 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34876, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,763 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,763 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34874, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:46,765 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34884, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,767 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,768 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34896, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,769 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:34914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-24 09:00:46,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:46,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:46,774 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:46,774 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:46,774 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:46,774 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:46,774 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,775 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:46,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:46,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 09:00:46,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,782 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,782 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:46,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:46,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:46,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:46,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:46,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:46,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:46,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:46,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:46,782 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,782 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:46,782 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:46,782 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,782 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:46,783 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:46,783 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:46,783 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:46,783 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:46,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:46,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,788 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:46,788 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:46,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:46,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fd6ca6d[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:46,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:46,790 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326906792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326906793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326846790, max timestamp=1682326846795Num uniques 859; 2023-04-24 09:00:46,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:46,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682326846790, max timestamp=1682326846795Num uniques 859; 2023-04-24 09:00:46,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:46,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326846790, max timestamp=1682326846795Num uniques 859; 2023-04-24 09:00:46,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:46,797 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:46,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:46,797 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:46,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:46,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:46,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:46,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:46,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:46,807 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:46,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:46,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:46,807 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:46,807 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:46,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326906809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326906810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326906811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:46,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:46,814 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:46,814 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,821 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,821 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:46,821 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:46,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:46,821 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,821 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:46,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:46,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:46,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:46,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:46,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:46,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,822 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:46,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,822 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:46,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,822 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326906897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326906897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326906912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326906913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:46,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326906914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,922 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:46,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:46,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:46,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:46,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:46,925 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:46,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:46,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:46,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:46,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,925 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:46,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:46,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:46,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:46,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:46,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:46,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:46,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:46,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:46,931 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:46,931 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,931 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:46,931 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:46,932 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:46,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:46,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:46,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:46,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:46,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:46,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:46,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:46,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c2e0096[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:46,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:46,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,946 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:46,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:46,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:46,947 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:46,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:46,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:46,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:46,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:46,956 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:46,957 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:46,957 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:46,957 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:46,957 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:46,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:46,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:46,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:46,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:46,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:46,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:46,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:46,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:46,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:46,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:46,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:46,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:46,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:46,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:46,972 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:46,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:46,972 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:47,072 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:47,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:47,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:47,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:47,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:47,075 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:47,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:47,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:47,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326907099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326907101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326907115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326907115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326907118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,121 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,121 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,121 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,121 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:47,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:47,122 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,122 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:47,122 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:47,122 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,122 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:47,123 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:47,138 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,138 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,138 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,138 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:47,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37487d2b[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:47,140 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:47,140 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,146 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:47,147 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,147 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:47,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:47,147 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:47,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:47,155 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:47,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:47,157 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:47,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:47,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:47,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:47,157 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:47,163 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,163 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,163 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,163 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:47,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:47,171 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:47,172 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:47,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,172 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,193 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424e7db7e776a974b1582d7ea4dba67ea9f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e7db7e776a974b1582d7ea4dba67ea9f 2023-04-24 09:00:47,272 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:47,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:47,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:47,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:47,275 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:47,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:47,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:47,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,280 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,280 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,280 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,280 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:47,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:47,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:47,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:47,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:47,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:47,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:47,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:47,282 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,282 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:47,282 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:47,282 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:47,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:47,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:47,282 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:47,288 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,288 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,288 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,288 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:47,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2884e699[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:47,289 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:47,290 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,296 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:47,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:47,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:47,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:47,305 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:47,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:47,306 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:47,306 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:47,306 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:47,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:47,307 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:47,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,313 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,314 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,321 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:47,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:47,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:47,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,322 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:47,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,322 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:47,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326907401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326907405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326907418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326907419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326907420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,422 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:47,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:47,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:47,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:47,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:47,425 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:47,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:47,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:47,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:47,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:47,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:47,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:47,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:47,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:47,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:47,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:47,438 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:47,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f8c446[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:47,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:47,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,446 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:47,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:47,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:47,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:47,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:47,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:47,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:47,457 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:47,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:47,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:47,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:47,457 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:47,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,464 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,472 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:47,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:47,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:47,472 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:47,472 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:47,572 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:47,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:47,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:47,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:47,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:47,575 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:47,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:47,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:47,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,580 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,580 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,580 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,580 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:47,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:47,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:47,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:47,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:47,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:47,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:47,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:47,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:47,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:47,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:47,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:47,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:47,598 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/ed2d9c2fdfbe49c6b08f325243877deb 2023-04-24 09:00:47,605 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:47,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6e445f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:47,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:47,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,613 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:47,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:47,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:47,613 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,613 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:47,621 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:47,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,621 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,622 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:47,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:47,623 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:47,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:47,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:47,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:47,623 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:47,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,630 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:47,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:47,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,638 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,638 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:47,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,638 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:47,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,739 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:47,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:47,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:47,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:47,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:47,741 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:47,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:47,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:47,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,742 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,746 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,746 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:47,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:47,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:47,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:47,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:47,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:47,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:47,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:47,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:47,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19bbcdfc[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:47,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:47,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:47,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:47,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:47,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:47,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,772 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:47,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:47,773 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:47,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:47,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:47,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:47,773 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:47,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,780 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,788 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:47,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:47,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:47,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,789 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:47,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,789 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:47,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,889 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:47,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:47,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:47,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:47,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:47,891 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:47,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:47,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:47,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,896 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:47,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:47,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:47,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:47,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:47,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:47,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:47,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:47,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1604507c[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:47,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:47,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326907906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326907909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:47,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:47,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:47,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:47,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:47,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,922 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326907921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326907922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,923 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:47,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:47,924 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:47,924 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:47,924 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:47,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:47,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:47,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326907923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:47,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:47,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,931 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:47,938 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:47,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:47,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:47,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:47,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:47,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:47,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:47,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:47,939 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,939 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:47,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:47,939 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:47,939 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:47,939 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/14703ee64ec4446dac3c260802d9847e 2023-04-24 09:00:48,039 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:48,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:48,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:48,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:48,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:48,041 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:48,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:48,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:48,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:48,042 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,042 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,046 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,047 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:48,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:48,047 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,047 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:48,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:48,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:48,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:48,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:48,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:48,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:48,048 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:48,055 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,055 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,055 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,055 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:48,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c97c0e[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:48,056 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:48,056 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,063 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:48,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:48,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:48,063 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:48,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:48,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:48,089 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:48,089 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:48,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:48,090 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:48,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:48,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:48,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:48,091 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:48,103 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,103 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,103 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,103 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,104 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:48,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:48,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:48,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:48,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:48,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,113 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:48,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:48,113 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:48,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,114 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:48,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:48,114 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,114 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,114 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,214 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:48,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:48,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:48,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:48,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:48,217 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:48,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:48,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:48,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:48,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,217 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,221 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,221 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:48,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:48,222 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,222 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:48,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:48,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:48,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:48,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:48,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:48,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:48,223 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:48,223 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:48,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,230 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,230 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,230 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:48,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e9a8cd4[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:48,231 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:48,231 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,238 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:48,238 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:48,238 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,238 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:48,238 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:48,246 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:48,246 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,246 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,246 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:48,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:48,248 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:48,248 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:48,248 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:48,248 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:48,248 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:48,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,255 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,256 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:48,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:48,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:48,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:48,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:48,263 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:48,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:48,263 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:48,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:48,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,264 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:48,264 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,364 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:48,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:48,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:48,367 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:48,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:48,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:48,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:48,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,367 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,371 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,371 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:48,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:48,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,372 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:48,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:48,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:48,373 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:48,373 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:48,373 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:48,373 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:48,373 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:48,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:48,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@598976d6[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:48,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:48,381 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,388 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,388 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:48,388 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:48,388 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,388 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:48,388 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:48,396 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,396 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:48,397 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,397 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:48,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:48,398 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:48,399 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:48,399 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:48,399 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:48,399 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:48,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:48,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:48,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:48,405 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,406 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,413 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/9ee13b1bd9e54876a36bd777132ac465 2023-04-24 09:00:48,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:48,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:48,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:48,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:48,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:48,413 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:48,413 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:48,413 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:48,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:48,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:48,414 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,414 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,414 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:48,414 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/ed2d9c2fdfbe49c6b08f325243877deb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ed2d9c2fdfbe49c6b08f325243877deb 2023-04-24 09:00:48,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ed2d9c2fdfbe49c6b08f325243877deb, entries=150, sequenceid=15, filesize=25.1 K 2023-04-24 09:00:48,419 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/14703ee64ec4446dac3c260802d9847e as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/14703ee64ec4446dac3c260802d9847e 2023-04-24 09:00:48,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/14703ee64ec4446dac3c260802d9847e, entries=150, sequenceid=15, filesize=11.6 K 2023-04-24 09:00:48,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/9ee13b1bd9e54876a36bd777132ac465 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9ee13b1bd9e54876a36bd777132ac465 2023-04-24 09:00:48,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9ee13b1bd9e54876a36bd777132ac465, entries=150, sequenceid=15, filesize=11.6 K 2023-04-24 09:00:48,425 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d6e3ad0fadc5b98d862d4692cd3586da in 1644ms, sequenceid=15, compaction requested=false 2023-04-24 09:00:48,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:48,514 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:48,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:48,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:48,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:48,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:48,516 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:48,516 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:48,516 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:48,516 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:48,517 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,517 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,521 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:48,521 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:48,522 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,522 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:48,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:48,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:48,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:48,523 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:48,523 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:48,523 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:48,523 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:48,523 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:48,523 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:48,523 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:48,523 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-24 09:00:48,523 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:48,523 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:48,523 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:48,523 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:48,524 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:48,524 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:48,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:48,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:48,910 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-24 09:00:48,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:48,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326908927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:48,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326908928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:48,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326908928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:48,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326908929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,934 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424ec4ec137765b4c0f92080e0250706fc0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ec4ec137765b4c0f92080e0250706fc0 2023-04-24 09:00:48,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:48,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326908934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:48,938 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/01e907ae82bf4d27af9aa640948106b7 2023-04-24 09:00:49,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326909029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326909029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326909232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326909233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,349 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/7ddaffa3809f4fd3bc9bbebbb34aee12 2023-04-24 09:00:49,403 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/0f6015dd8d3441ca881c36fdfbc4fb0f 2023-04-24 09:00:49,407 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/01e907ae82bf4d27af9aa640948106b7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/01e907ae82bf4d27af9aa640948106b7 2023-04-24 09:00:49,409 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/01e907ae82bf4d27af9aa640948106b7, entries=150, sequenceid=50, filesize=25.1 K 2023-04-24 09:00:49,410 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/7ddaffa3809f4fd3bc9bbebbb34aee12 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7ddaffa3809f4fd3bc9bbebbb34aee12 2023-04-24 09:00:49,412 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7ddaffa3809f4fd3bc9bbebbb34aee12, entries=150, sequenceid=50, filesize=11.6 K 2023-04-24 09:00:49,413 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/0f6015dd8d3441ca881c36fdfbc4fb0f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0f6015dd8d3441ca881c36fdfbc4fb0f 2023-04-24 09:00:49,415 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0f6015dd8d3441ca881c36fdfbc4fb0f, entries=150, sequenceid=50, filesize=11.6 K 2023-04-24 09:00:49,416 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for d6e3ad0fadc5b98d862d4692cd3586da in 893ms, sequenceid=50, compaction requested=false 2023-04-24 09:00:49,416 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:49,416 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:49,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:49,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:49,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:49,416 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:49,421 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,421 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:49,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d3a5a2b[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:49,423 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:49,423 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,430 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:49,430 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:49,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:49,430 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:49,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:49,438 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:49,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:49,440 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:49,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:49,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:49,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:49,440 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:49,464 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,464 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,464 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,465 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,466 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,480 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:49,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:49,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:49,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:49,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:49,480 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:49,480 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:49,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:49,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:49,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 09:00:49,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:49,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:49,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:49,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:49,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:49,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:49,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326909558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326909559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,581 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:49,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:49,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:49,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:49,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:49,584 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:49,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:49,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:49,584 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:49,585 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,585 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,594 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:49,594 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:49,595 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,595 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:49,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:49,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:49,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:49,595 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:49,595 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:49,595 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:49,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:49,596 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:49,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:49,596 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:49,596 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:49,596 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:49,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:49,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:49,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:49,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:49,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,605 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:49,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2759628e[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:49,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:49,606 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,613 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:49,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:49,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:49,613 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,613 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:49,621 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:49,621 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:49,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,621 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:49,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:49,623 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:49,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:49,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:49,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:49,623 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:49,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,630 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:49,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:49,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:49,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:49,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,638 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:49,638 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:49,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:49,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:49,639 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,638 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:49,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326909662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326909662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,739 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:49,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:49,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:49,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:49,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:49,741 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:49,742 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:49,742 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:49,742 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:49,742 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,742 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,746 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,746 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:49,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:49,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:49,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:49,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:49,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:49,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:49,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:49,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:49,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:49,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:49,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:49,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6fa69d[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:49,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:49,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:49,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:49,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:49,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:49,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:49,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,772 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:49,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:49,773 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:49,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:49,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:49,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:49,773 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:49,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:49,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,780 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,788 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:49,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:49,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:49,788 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:49,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:49,788 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:49,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:49,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:49,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326909864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:49,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326909865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,889 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:49,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:49,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:49,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:49,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:49,891 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:49,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:49,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:49,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:49,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,896 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,896 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:49,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:49,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:49,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:49,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:49,897 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:49,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:49,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:49,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:49,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:49,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:49,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc25220[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:49,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:49,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:49,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:49,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:49,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:49,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:49,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,922 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:49,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:49,923 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:49,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:49,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:49,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:49,923 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:49,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:49,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:49,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,931 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,946 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424c9459e1066104c388ed74e514cc0cdd4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c9459e1066104c388ed74e514cc0cdd4 2023-04-24 09:00:49,950 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/a790c0c09f6a4c309f80a3463a56fa7f 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:49,957 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:49,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/38c641fc75f6449a95001db498cd01ab 2023-04-24 09:00:49,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:49,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:49,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:49,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:49,957 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:49,957 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:49,957 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,057 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:50,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:50,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:50,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:50,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:50,060 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:50,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:50,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:50,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:50,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,060 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:50,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:50,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:50,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:50,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:50,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:50,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:50,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:50,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:50,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:50,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:50,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:50,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:50,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:50,140 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:50,140 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:50,140 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:50,140 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:50,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:50,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4016a4f1[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:50,141 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:50,141 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:50,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326910168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326910168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,271 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:50,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:50,272 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,272 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:50,272 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:50,272 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:50,272 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:50,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/04de6d7ee32247ffa1fdb48b61fffd4a 2023-04-24 09:00:50,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/a790c0c09f6a4c309f80a3463a56fa7f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/a790c0c09f6a4c309f80a3463a56fa7f 2023-04-24 09:00:50,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/a790c0c09f6a4c309f80a3463a56fa7f, entries=200, sequenceid=63, filesize=31.9 K 2023-04-24 09:00:50,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/38c641fc75f6449a95001db498cd01ab as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/38c641fc75f6449a95001db498cd01ab 2023-04-24 09:00:50,380 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:50,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:50,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:50,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/38c641fc75f6449a95001db498cd01ab, entries=150, sequenceid=63, filesize=11.6 K 2023-04-24 09:00:50,381 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:50,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/04de6d7ee32247ffa1fdb48b61fffd4a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/04de6d7ee32247ffa1fdb48b61fffd4a 2023-04-24 09:00:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/04de6d7ee32247ffa1fdb48b61fffd4a, entries=150, sequenceid=63, filesize=11.6 K 2023-04-24 09:00:50,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d6e3ad0fadc5b98d862d4692cd3586da in 865ms, sequenceid=63, compaction requested=true 2023-04-24 09:00:50,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:50,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:50,402 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:50,402 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:50,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:50,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:50,403 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:50,403 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:50,403 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/A is initiating minor compaction (all files) 2023-04-24 09:00:50,403 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/B is initiating minor compaction (all files) 2023-04-24 09:00:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,403 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/A in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:50,403 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/B in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:50,404 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ed2d9c2fdfbe49c6b08f325243877deb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/01e907ae82bf4d27af9aa640948106b7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/a790c0c09f6a4c309f80a3463a56fa7f] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=82.2 K 2023-04-24 09:00:50,404 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/14703ee64ec4446dac3c260802d9847e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7ddaffa3809f4fd3bc9bbebbb34aee12, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/38c641fc75f6449a95001db498cd01ab] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=34.8 K 2023-04-24 09:00:50,404 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 14703ee64ec4446dac3c260802d9847e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682326846774 2023-04-24 09:00:50,404 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting ed2d9c2fdfbe49c6b08f325243877deb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682326846774 2023-04-24 09:00:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:50,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:50,404 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:50,404 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:50,404 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:50,404 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7ddaffa3809f4fd3bc9bbebbb34aee12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682326846801 2023-04-24 09:00:50,404 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:50,404 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:50,405 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 01e907ae82bf4d27af9aa640948106b7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682326846801 2023-04-24 09:00:50,405 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 38c641fc75f6449a95001db498cd01ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682326848922 2023-04-24 09:00:50,405 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting a790c0c09f6a4c309f80a3463a56fa7f, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682326848922 2023-04-24 09:00:50,421 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,421 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:50,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:50,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:50,422 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:50,423 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:50,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:50,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:50,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:50,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:50,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:50,430 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:50,431 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:50,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:50,432 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#B#compaction#439 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:50,453 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#A#compaction#438 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:50,531 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:50,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:50,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:50,533 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:50,533 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:50,533 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:50,533 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:50,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,534 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:50,538 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:50,538 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:50,538 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,538 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:50,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:50,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:50,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:50,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:50,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:50,539 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:50,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:50,540 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:50,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:50,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:50,540 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-24 09:00:50,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:50,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:50,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:50,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:50,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:50,540 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:50,552 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424dc363da8b09f4dfd80fb0dd70598db53 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424dc363da8b09f4dfd80fb0dd70598db53 2023-04-24 09:00:50,557 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/8854b96e331646a595077c8af3f083b3 2023-04-24 09:00:50,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:50,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:50,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326910684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326910685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326910787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326910788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,859 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/19a8b684a95f434b9b8dc51b369c4909 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/19a8b684a95f434b9b8dc51b369c4909 2023-04-24 09:00:50,862 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/B of d6e3ad0fadc5b98d862d4692cd3586da into 19a8b684a95f434b9b8dc51b369c4909(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:50,862 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:50,862 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/B, priority=13, startTime=1682326850402; duration=0sec 2023-04-24 09:00:50,862 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:50,862 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:50,863 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:50,863 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/C is initiating minor compaction (all files) 2023-04-24 09:00:50,863 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/C in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:50,863 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9ee13b1bd9e54876a36bd777132ac465, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0f6015dd8d3441ca881c36fdfbc4fb0f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/04de6d7ee32247ffa1fdb48b61fffd4a] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=34.8 K 2023-04-24 09:00:50,863 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 9ee13b1bd9e54876a36bd777132ac465, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682326846774 2023-04-24 09:00:50,863 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 0f6015dd8d3441ca881c36fdfbc4fb0f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682326846801 2023-04-24 09:00:50,864 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 04de6d7ee32247ffa1fdb48b61fffd4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682326848922 2023-04-24 09:00:50,868 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#C#compaction#442 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:50,878 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/7535703bd9c14189be2b0fa353a4c1ad as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/7535703bd9c14189be2b0fa353a4c1ad 2023-04-24 09:00:50,882 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/C of d6e3ad0fadc5b98d862d4692cd3586da into 7535703bd9c14189be2b0fa353a4c1ad(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:50,882 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:50,882 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/C, priority=13, startTime=1682326850403; duration=0sec 2023-04-24 09:00:50,882 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:50,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326910936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326910936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,938 DEBUG [Thread-3270] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:50,939 DEBUG [Thread-3272] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:50,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326910947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,950 DEBUG [Thread-3268] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:50,969 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/37efcf69aa7944fb8960e21d0927a370 2023-04-24 09:00:50,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326910990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:50,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:50,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326910990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,282 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/32914c9710a64b139343f387b831620f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/32914c9710a64b139343f387b831620f 2023-04-24 09:00:51,285 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/A of d6e3ad0fadc5b98d862d4692cd3586da into 32914c9710a64b139343f387b831620f(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:51,285 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:51,285 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/A, priority=13, startTime=1682326850402; duration=0sec 2023-04-24 09:00:51,285 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:51,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:51,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326911293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:51,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326911293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,377 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/5f0b42148f064de49876dafee8697a62 2023-04-24 09:00:51,380 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/8854b96e331646a595077c8af3f083b3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8854b96e331646a595077c8af3f083b3 2023-04-24 09:00:51,383 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8854b96e331646a595077c8af3f083b3, entries=150, sequenceid=82, filesize=25.1 K 2023-04-24 09:00:51,384 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/37efcf69aa7944fb8960e21d0927a370 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/37efcf69aa7944fb8960e21d0927a370 2023-04-24 09:00:51,386 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/37efcf69aa7944fb8960e21d0927a370, entries=150, sequenceid=82, filesize=11.6 K 2023-04-24 09:00:51,386 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/5f0b42148f064de49876dafee8697a62 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5f0b42148f064de49876dafee8697a62 2023-04-24 09:00:51,389 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5f0b42148f064de49876dafee8697a62, entries=150, sequenceid=82, filesize=11.6 K 2023-04-24 09:00:51,390 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d6e3ad0fadc5b98d862d4692cd3586da in 850ms, sequenceid=82, compaction requested=false 2023-04-24 09:00:51,390 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:51,390 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:51,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:51,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:51,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:51,390 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:51,421 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,421 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:51,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:51,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:51,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:51,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c37e36f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:51,423 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:51,423 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:51,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:51,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:51,430 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:51,430 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:51,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:51,430 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:51,438 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:51,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:51,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:51,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:51,440 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:51,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:51,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:51,440 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:51,440 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:51,446 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,446 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,446 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:51,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:51,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:51,448 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:51,455 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:51,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:51,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:51,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:51,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:51,455 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:51,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:51,455 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:51,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,555 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:51,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:51,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:51,558 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:51,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:51,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:51,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:51,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,558 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:51,563 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:51,563 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:51,563 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,563 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:51,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:51,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:51,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:51,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:51,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:51,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:51,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:51,564 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:51,565 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:51,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:51,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:51,565 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-24 09:00:51,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:51,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:51,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:51,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:51,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:51,565 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:51,579 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042445318fe8b0e24d04b6c8f1d2d9a1ec59 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042445318fe8b0e24d04b6c8f1d2d9a1ec59 2023-04-24 09:00:51,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:51,796 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:51,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:51,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326911814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:51,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326911815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326911917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:51,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326911918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:51,991 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/852d85939025427bb7e0d3d80c7d7d46 2023-04-24 09:00:51,998 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/76665c93fe5545528b8c0e9743389524 2023-04-24 09:00:52,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:52,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326912121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326912122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,405 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/62be3ce3a7674888830178f5db28b47c 2023-04-24 09:00:52,408 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/852d85939025427bb7e0d3d80c7d7d46 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/852d85939025427bb7e0d3d80c7d7d46 2023-04-24 09:00:52,411 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/852d85939025427bb7e0d3d80c7d7d46, entries=150, sequenceid=98, filesize=25.1 K 2023-04-24 09:00:52,411 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/76665c93fe5545528b8c0e9743389524 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/76665c93fe5545528b8c0e9743389524 2023-04-24 09:00:52,416 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/76665c93fe5545528b8c0e9743389524, entries=150, sequenceid=98, filesize=11.6 K 2023-04-24 09:00:52,417 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/62be3ce3a7674888830178f5db28b47c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/62be3ce3a7674888830178f5db28b47c 2023-04-24 09:00:52,421 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/62be3ce3a7674888830178f5db28b47c, entries=150, sequenceid=98, filesize=11.6 K 2023-04-24 09:00:52,422 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d6e3ad0fadc5b98d862d4692cd3586da in 857ms, sequenceid=98, compaction requested=true 2023-04-24 09:00:52,422 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:52,422 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:52,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:52,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:52,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:52,422 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:52,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:52,427 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-24 09:00:52,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:52,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:52,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:52,428 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:52,428 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:52,428 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:52,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:52,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326912438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:52,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326912440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,480 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,480 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,480 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,480 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:52,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7ce33a[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:52,481 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:52,481 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,488 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,488 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:52,488 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:52,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,488 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:52,488 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:52,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:52,496 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,496 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:52,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:52,498 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:52,498 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:52,498 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:52,498 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:52,498 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:52,505 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,505 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,505 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:52,505 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,506 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,532 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,532 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:52,532 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:52,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:52,532 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:52,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:52,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:52,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:52,532 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,533 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:52,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,533 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:52,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:52,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,533 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:52,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326912541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326912543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,633 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:52,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:52,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:52,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:52,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:52,636 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:52,636 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:52,636 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:52,636 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:52,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,637 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,646 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:52,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:52,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,647 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:52,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:52,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:52,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:52,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:52,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:52,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:52,648 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:52,648 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:52,655 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,655 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,655 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,655 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:52,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51c2f5c7[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:52,656 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:52,656 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,663 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,663 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:52,663 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:52,663 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,663 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:52,663 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:52,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:52,671 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,671 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:52,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:52,673 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:52,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:52,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:52,674 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:52,674 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:52,680 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,680 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,680 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:52,680 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,681 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:52,688 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:52,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:52,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:52,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,688 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:52,688 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:52,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:52,689 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:52,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:52,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326912743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:52,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326912745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,789 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:52,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:52,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:52,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:52,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:52,792 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:52,792 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:52,792 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:52,792 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:52,792 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,792 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,796 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,796 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:52,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:52,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:52,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:52,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:52,798 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:52,798 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:52,798 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:52,798 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:52,798 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:52,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,805 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:52,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2486e222[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:52,806 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:52,806 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,813 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,813 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:52,813 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:52,813 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:52,813 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,813 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:52,821 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:52,821 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:52,821 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,821 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:52,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:52,823 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:52,823 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:52,823 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:52,824 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:52,824 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:52,830 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,830 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,830 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,830 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,831 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,837 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304240bfc169decc74e4183356fe68da2b0b0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240bfc169decc74e4183356fe68da2b0b0 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:52,838 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,838 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:52,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:52,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:52,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:52,839 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:52,839 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:52,939 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:52,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:52,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:52,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:52,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:52,942 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:52,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:52,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:52,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:52,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,942 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,946 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:52,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:52,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:52,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:52,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:52,948 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:52,948 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:52,948 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:52,948 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:52,948 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:52,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:52,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37b95146[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:52,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:52,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,963 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:52,963 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,963 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:52,963 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:52,963 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:52,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,971 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,971 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:52,972 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:52,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:52,973 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:52,973 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:52,973 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:52,973 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:52,973 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:52,980 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,980 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:52,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,980 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:52,980 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:52,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,981 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,988 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,988 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,988 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:52,988 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:52,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:52,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:52,988 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:52,988 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:52,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:52,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:52,988 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:52,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:52,988 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,988 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:52,989 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:52,989 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:52,989 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:53,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326913045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:53,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326913048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,089 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:53,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:53,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:53,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:53,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:53,091 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:53,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:53,091 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:53,092 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,092 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,096 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,096 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,097 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,097 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:53,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:53,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:53,098 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,098 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:53,098 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:53,098 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:53,098 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:53,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,130 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,131 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,131 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:53,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c8701b8[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:53,132 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:53,132 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,144 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,144 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,144 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:53,144 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:53,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:53,145 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:53,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:53,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,155 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,155 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:53,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:53,157 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:53,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:53,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:53,157 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:53,157 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:53,163 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,163 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,163 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,163 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,164 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,171 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:53,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:53,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:53,171 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,172 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:53,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,172 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:53,172 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,242 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/b2871a2964fc4a5b977f3ed3c1628b0b 2023-04-24 09:00:53,272 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:53,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:53,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:53,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:53,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:53,275 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:53,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:53,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:53,275 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,276 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,280 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,280 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,280 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,280 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:53,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:53,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:53,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:53,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:53,281 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:53,281 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:53,288 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,288 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,288 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,288 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:53,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aadad22[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:53,289 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:53,289 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,296 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:53,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,297 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:53,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:53,297 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:53,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:53,305 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,305 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:53,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:53,307 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:53,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:53,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:53,307 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:53,307 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:53,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,313 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,313 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,315 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:53,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:53,321 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:53,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,322 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:53,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,322 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:53,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,322 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,422 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:53,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:53,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:53,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:53,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:53,425 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:53,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:53,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:53,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,425 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,430 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,430 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:53,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:53,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:53,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:53,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:53,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:53,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:53,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:53,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:53,431 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:53,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:53,431 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,432 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:53,432 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:53,438 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,438 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:53,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3929cda7[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:53,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:53,439 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,446 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:53,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:53,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,447 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:53,447 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:53,455 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,455 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:53,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:53,457 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:53,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:53,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:53,457 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:53,457 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:53,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,464 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:53,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:53,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,471 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,472 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:53,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:53,471 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,472 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:53,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,472 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326913550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:53,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326913553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,572 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:53,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:53,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:53,575 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:53,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:53,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:53,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,575 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,580 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,580 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,580 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,580 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:53,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:53,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:53,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:53,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:53,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:53,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:53,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:53,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:53,581 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:53,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:53,581 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:53,582 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:53,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,588 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,588 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,588 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:53,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e798a5[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:53,590 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:53,590 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,614 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,615 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,615 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:53,615 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:53,615 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:53,615 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:53,627 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:53,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,628 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:53,628 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:53,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:53,630 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:53,630 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:53,630 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:53,630 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:53,630 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:53,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,638 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,639 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,646 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,646 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,646 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:53,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:53,646 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,647 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:53,647 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:53,646 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,646 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,647 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,649 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/5358e906f9884a6d8a3eac72be8b0cc0 2023-04-24 09:00:53,747 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:53,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:53,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:53,750 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:53,750 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,750 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:53,750 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:53,750 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,750 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,755 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,755 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,755 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:53,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:53,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:53,756 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,756 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:53,756 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:53,756 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:53,756 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:53,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:53,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:53,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1039cb40[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:53,764 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:53,764 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:53,771 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,772 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:53,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:53,772 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:53,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,789 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:53,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:53,791 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:53,791 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:53,791 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:53,791 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:53,791 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:53,796 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,796 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,796 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,797 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,798 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,805 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:53,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:53,805 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,805 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:53,805 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:53,906 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:53,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:53,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:53,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:53,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:53,908 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:53,908 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,908 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:53,908 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:53,909 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,909 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:53,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:53,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:53,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:53,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:53,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:53,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:53,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:53,914 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:53,914 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,915 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:53,915 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:53,915 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:53,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:53,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:53,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:53,915 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:53,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,922 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,922 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:53,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57baa0d8[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:53,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:53,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:53,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:53,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:53,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:53,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:53,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:53,940 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:53,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:53,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:53,940 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:53,940 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:53,946 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,946 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:53,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,946 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,947 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,955 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:53,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:53,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:53,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:53,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:53,955 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:53,955 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:53,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:53,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:54,055 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:54,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/4862d286b3f24812aa0c4fbc490a57e3 2023-04-24 09:00:54,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:54,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:54,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:54,058 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:54,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:54,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:54,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:54,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,059 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:54,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/b2871a2964fc4a5b977f3ed3c1628b0b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/b2871a2964fc4a5b977f3ed3c1628b0b 2023-04-24 09:00:54,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/b2871a2964fc4a5b977f3ed3c1628b0b, entries=150, sequenceid=118, filesize=25.1 K 2023-04-24 09:00:54,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/5358e906f9884a6d8a3eac72be8b0cc0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5358e906f9884a6d8a3eac72be8b0cc0 2023-04-24 09:00:54,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,063 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:54,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:54,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:54,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:54,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:54,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:54,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:54,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:54,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:54,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:54,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:54,064 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:54,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:54,064 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:54,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:54,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:54,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:54,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:54,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:54,065 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:54,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5358e906f9884a6d8a3eac72be8b0cc0, entries=150, sequenceid=118, filesize=11.6 K 2023-04-24 09:00:54,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/4862d286b3f24812aa0c4fbc490a57e3 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/4862d286b3f24812aa0c4fbc490a57e3 2023-04-24 09:00:54,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/4862d286b3f24812aa0c4fbc490a57e3, entries=150, sequenceid=118, filesize=11.6 K 2023-04-24 09:00:54,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d6e3ad0fadc5b98d862d4692cd3586da in 1642ms, sequenceid=118, compaction requested=true 2023-04-24 09:00:54,069 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:54,070 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:54,070 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:54,070 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:54,070 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:54,070 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:54,070 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103087 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:54,070 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:54,071 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/A is initiating minor compaction (all files) 2023-04-24 09:00:54,071 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/B is initiating minor compaction (all files) 2023-04-24 09:00:54,071 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/A in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:54,071 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/B in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:54,071 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/32914c9710a64b139343f387b831620f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8854b96e331646a595077c8af3f083b3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/852d85939025427bb7e0d3d80c7d7d46, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/b2871a2964fc4a5b977f3ed3c1628b0b] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=100.7 K 2023-04-24 09:00:54,071 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/19a8b684a95f434b9b8dc51b369c4909, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/37efcf69aa7944fb8960e21d0927a370, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/76665c93fe5545528b8c0e9743389524, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5358e906f9884a6d8a3eac72be8b0cc0] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=46.5 K 2023-04-24 09:00:54,071 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 32914c9710a64b139343f387b831620f, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682326848922 2023-04-24 09:00:54,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,071 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:54,071 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 8854b96e331646a595077c8af3f083b3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326849556 2023-04-24 09:00:54,071 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 19a8b684a95f434b9b8dc51b369c4909, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682326848922 2023-04-24 09:00:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:54,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:54,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:54,072 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 852d85939025427bb7e0d3d80c7d7d46, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326850682 2023-04-24 09:00:54,072 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 37efcf69aa7944fb8960e21d0927a370, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326849556 2023-04-24 09:00:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:54,072 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting b2871a2964fc4a5b977f3ed3c1628b0b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682326851813 2023-04-24 09:00:54,072 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 76665c93fe5545528b8c0e9743389524, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326850682 2023-04-24 09:00:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:54,072 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5358e906f9884a6d8a3eac72be8b0cc0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682326851813 2023-04-24 09:00:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:54,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:54,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20890b7d[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:54,073 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:54,073 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:54,078 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#B#compaction#451 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:54,080 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#A#compaction#450 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:54,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:54,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:54,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:54,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:54,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:54,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:54,086 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/5843ec43c5c940049c7c6610e453062a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5843ec43c5c940049c7c6610e453062a 2023-04-24 09:00:54,090 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/B of d6e3ad0fadc5b98d862d4692cd3586da into 5843ec43c5c940049c7c6610e453062a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:54,090 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:54,090 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/B, priority=12, startTime=1682326854070; duration=0sec 2023-04-24 09:00:54,090 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:54,090 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:00:54,091 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:00:54,091 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/C is initiating minor compaction (all files) 2023-04-24 09:00:54,091 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/C in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:54,091 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/7535703bd9c14189be2b0fa353a4c1ad, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5f0b42148f064de49876dafee8697a62, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/62be3ce3a7674888830178f5db28b47c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/4862d286b3f24812aa0c4fbc490a57e3] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=46.5 K 2023-04-24 09:00:54,091 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7535703bd9c14189be2b0fa353a4c1ad, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1682326848922 2023-04-24 09:00:54,092 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5f0b42148f064de49876dafee8697a62, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682326849556 2023-04-24 09:00:54,092 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 62be3ce3a7674888830178f5db28b47c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1682326850682 2023-04-24 09:00:54,092 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 4862d286b3f24812aa0c4fbc490a57e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682326851813 2023-04-24 09:00:54,096 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#C#compaction#452 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:54,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:54,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:54,107 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,107 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:54,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:54,109 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:54,109 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:54,109 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:54,109 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:54,109 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:54,121 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,121 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:54,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,121 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:54,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:54,122 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:54,123 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:54,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:54,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:54,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:54,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:54,130 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:54,130 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:54,130 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:54,130 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,230 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:54,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:54,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:54,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:54,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:54,233 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:54,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:54,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:54,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:54,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,233 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:54,238 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:54,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:54,238 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,238 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:54,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:54,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:54,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:54,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:54,239 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:54,239 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:54,239 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:54,239 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:54,239 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:54,239 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:54,239 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:54,240 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 09:00:54,240 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:54,240 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:54,240 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:54,240 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:54,240 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:54,240 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:54,494 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/f1fcfe1c8c5c4ab089b5955896279550 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/f1fcfe1c8c5c4ab089b5955896279550 2023-04-24 09:00:54,497 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/A of d6e3ad0fadc5b98d862d4692cd3586da into f1fcfe1c8c5c4ab089b5955896279550(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:54,497 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:54,497 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/A, priority=12, startTime=1682326854070; duration=0sec 2023-04-24 09:00:54,497 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:54,502 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/2df0aea543394b70af88c5d1847da837 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/2df0aea543394b70af88c5d1847da837 2023-04-24 09:00:54,505 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/C of d6e3ad0fadc5b98d862d4692cd3586da into 2df0aea543394b70af88c5d1847da837(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:54,505 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:54,505 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/C, priority=12, startTime=1682326854070; duration=0sec 2023-04-24 09:00:54,506 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:54,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:54,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:54,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326914577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326914578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,649 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042467701ed80d1b4fc981e8f8aea4ffa89a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042467701ed80d1b4fc981e8f8aea4ffa89a 2023-04-24 09:00:54,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326914679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326914680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326914881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326914882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326914949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,951 DEBUG [Thread-3270] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326914962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,964 DEBUG [Thread-3272] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:54,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:54,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326914968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:54,970 DEBUG [Thread-3268] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:00:55,053 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/67303348e29f44beb4f914f73a4b2e37 2023-04-24 09:00:55,059 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/532e06f2e7094989b4354fbc5740e4b8 2023-04-24 09:00:55,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:55,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326915184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:55,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326915185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,467 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/5aa6cb8c4a4f48c0aa530466fea198dc 2023-04-24 09:00:55,470 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/67303348e29f44beb4f914f73a4b2e37 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/67303348e29f44beb4f914f73a4b2e37 2023-04-24 09:00:55,472 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/67303348e29f44beb4f914f73a4b2e37, entries=150, sequenceid=131, filesize=25.2 K 2023-04-24 09:00:55,473 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/532e06f2e7094989b4354fbc5740e4b8 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/532e06f2e7094989b4354fbc5740e4b8 2023-04-24 09:00:55,475 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/532e06f2e7094989b4354fbc5740e4b8, entries=150, sequenceid=131, filesize=11.7 K 2023-04-24 09:00:55,476 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/5aa6cb8c4a4f48c0aa530466fea198dc as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5aa6cb8c4a4f48c0aa530466fea198dc 2023-04-24 09:00:55,478 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5aa6cb8c4a4f48c0aa530466fea198dc, entries=150, sequenceid=131, filesize=11.7 K 2023-04-24 09:00:55,479 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d6e3ad0fadc5b98d862d4692cd3586da in 1240ms, sequenceid=131, compaction requested=false 2023-04-24 09:00:55,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:55,479 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:55,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:55,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:55,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:55,479 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:55,601 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,601 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,601 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,601 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:55,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58538d79[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:55,602 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:55,602 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,613 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:55,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:55,613 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,613 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:55,613 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:55,621 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:55,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,621 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:55,621 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:55,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:55,623 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:55,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:55,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:55,623 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:55,623 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:55,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:55,630 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,631 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,638 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:55,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:55,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:55,639 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:55,638 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,639 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:55,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:55,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:55,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-24 09:00:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:55,690 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:55,690 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:55,690 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:55,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:55,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326915704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326915705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,739 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:55,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:55,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:55,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:55,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:55,741 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:55,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:55,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:55,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:55,741 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,742 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,746 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:55,746 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:55,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:55,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:55,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:55,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:55,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:55,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:55,748 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:55,748 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:55,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:55,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66e2ef8[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:55,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:55,756 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:55,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:55,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:55,763 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:55,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:55,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:55,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:55,773 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:55,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:55,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:55,773 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:55,773 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:55,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,780 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:55,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:55,788 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:55,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:55,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:55,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:55,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:55,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:55,788 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:55,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,789 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:55,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:55,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326915807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:55,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326915808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,889 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:55,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:55,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:55,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:55,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:55,891 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:55,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:55,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:55,891 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:55,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,896 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:55,896 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:55,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,897 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:55,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:55,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:55,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:55,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:55,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:55,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:55,898 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:55,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:55,905 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,905 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:55,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0f28d[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:55,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:55,906 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:55,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:55,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:55,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:55,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:55,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:55,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:55,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:55,923 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:55,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:55,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:55,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:55,923 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:55,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:55,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:55,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,931 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,958 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:55,958 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:55,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:55,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:55,958 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:55,958 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:55,958 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:55,958 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,958 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:55,959 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:55,959 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:55,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:55,959 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,959 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:55,959 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326916010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326916011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,059 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:56,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:56,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:56,061 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:56,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:56,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:56,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,061 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:56,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:56,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:56,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:56,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:56,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:56,073 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:56,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,080 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:56,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ceac389[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:56,081 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:56,081 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,088 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,088 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:56,088 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,088 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:56,088 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:56,088 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,096 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,096 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:56,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:56,098 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:56,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:56,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:56,098 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:56,098 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:56,103 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424e58aa4b1649441fdb3da68e279cd39ef to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e58aa4b1649441fdb3da68e279cd39ef 2023-04-24 09:00:56,105 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,105 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,105 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,105 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,107 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,113 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:56,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:56,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,113 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,113 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:56,114 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,114 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:56,114 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,214 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:56,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:56,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:56,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:56,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:56,223 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:56,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:56,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:56,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,223 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,230 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,230 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,230 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,230 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:56,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:56,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:56,231 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,231 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:56,231 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:56,231 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:56,231 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:56,238 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,238 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,238 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,238 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:56,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2816a328[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:56,239 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:56,239 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,246 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,246 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:56,246 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:56,246 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,247 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:56,247 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:56,255 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:56,255 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,255 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,255 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:56,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:56,256 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:56,256 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:56,256 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:56,257 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:56,257 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:56,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,263 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,263 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,264 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,271 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,271 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,271 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:56,271 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:56,271 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,271 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,272 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:56,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:56,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,272 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:56,272 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:56,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326916315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:56,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326916315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,372 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:56,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:56,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:56,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:56,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:56,375 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:56,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:56,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:56,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,375 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,380 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,380 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,380 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,380 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:56,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:56,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:56,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:56,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:56,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:56,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:56,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:56,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:56,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:56,381 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:56,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:56,381 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:56,382 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:56,388 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,388 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,388 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,388 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:56,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f78513c[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:56,389 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:56,390 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,396 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,396 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:56,397 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:56,396 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,397 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:56,397 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:56,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:56,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,405 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,405 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:56,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:56,407 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:56,407 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:56,407 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:56,407 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:56,407 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:56,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,413 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,413 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,414 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,442 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,442 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:56,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:56,442 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,442 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,442 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:56,442 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,442 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:56,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,442 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,443 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,443 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,443 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,443 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,507 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=154, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/bc48756af6ab4ff787d18f64d9786ab0 2023-04-24 09:00:56,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/eae114232f1148978c6fc2675f53773f 2023-04-24 09:00:56,542 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:56,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:56,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:56,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:56,545 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:56,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:56,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:56,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,545 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,579 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,579 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,579 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,579 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:56,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:56,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:56,580 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,580 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:56,580 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:56,580 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:56,580 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:56,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,588 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,588 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,588 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:56,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cbf84c4[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:56,589 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:56,590 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,596 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:56,596 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:56,596 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,597 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:56,597 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:56,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:56,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,605 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,605 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:56,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:56,607 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:56,607 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:56,607 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:56,607 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:56,607 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:56,613 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,613 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,613 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,613 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,614 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,621 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:56,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:56,621 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:56,622 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,622 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,622 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,622 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:56,622 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,722 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:56,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:56,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:56,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:56,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:56,725 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:56,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:56,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:56,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,725 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,730 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,730 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,730 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,730 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:56,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:56,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:56,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:56,731 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:56,731 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,731 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:56,731 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:56,731 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:56,731 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,731 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:56,731 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:56,731 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,731 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:56,731 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:56,731 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,732 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:56,732 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:56,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,738 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,738 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,738 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:56,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f3e251[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:56,739 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:56,740 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,746 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,746 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,746 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:56,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:56,747 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:56,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:56,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:56,755 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,755 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:56,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:56,757 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:56,757 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:56,757 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:56,757 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:56,757 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:56,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,763 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,763 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,764 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,771 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:56,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:56,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:56,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,772 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,772 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,772 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:56,772 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:56,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:56,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34874 deadline: 1682326916818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:56,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34858 deadline: 1682326916820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,872 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:56,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:56,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:56,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:56,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:56,875 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:56,875 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,875 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:56,875 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:56,876 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,876 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,886 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,886 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:56,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:56,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:56,888 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,888 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:56,888 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:56,888 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:56,888 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:56,896 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,897 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,897 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,897 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:56,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b610244[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:56,898 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:56,898 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:56,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:56,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,913 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:56,913 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:56,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:56,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:56,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:56,923 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:56,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:56,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:56,923 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:56,923 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:56,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/eecd9766a802452e9d78751101bfec09 2023-04-24 09:00:56,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:56,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/bc48756af6ab4ff787d18f64d9786ab0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/bc48756af6ab4ff787d18f64d9786ab0 2023-04-24 09:00:56,934 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/bc48756af6ab4ff787d18f64d9786ab0, entries=150, sequenceid=154, filesize=25.3 K 2023-04-24 09:00:56,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/eae114232f1148978c6fc2675f53773f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/eae114232f1148978c6fc2675f53773f 2023-04-24 09:00:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:56,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,936 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:56,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/eae114232f1148978c6fc2675f53773f, entries=150, sequenceid=154, filesize=11.8 K 2023-04-24 09:00:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:56,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/eecd9766a802452e9d78751101bfec09 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/eecd9766a802452e9d78751101bfec09 2023-04-24 09:00:56,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/eecd9766a802452e9d78751101bfec09, entries=150, sequenceid=154, filesize=11.8 K 2023-04-24 09:00:56,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for d6e3ad0fadc5b98d862d4692cd3586da in 1262ms, sequenceid=154, compaction requested=true 2023-04-24 09:00:56,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:56,951 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:56,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:56,951 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:56,951 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:56,951 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:56,952 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/A is initiating minor compaction (all files) 2023-04-24 09:00:56,952 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/B is initiating minor compaction (all files) 2023-04-24 09:00:56,952 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/A in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,952 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/B in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:56,952 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/f1fcfe1c8c5c4ab089b5955896279550, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/67303348e29f44beb4f914f73a4b2e37, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/bc48756af6ab4ff787d18f64d9786ab0] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=75.9 K 2023-04-24 09:00:56,952 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5843ec43c5c940049c7c6610e453062a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/532e06f2e7094989b4354fbc5740e4b8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/eae114232f1148978c6fc2675f53773f] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=35.3 K 2023-04-24 09:00:56,952 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting f1fcfe1c8c5c4ab089b5955896279550, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682326851813 2023-04-24 09:00:56,953 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5843ec43c5c940049c7c6610e453062a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682326851813 2023-04-24 09:00:56,953 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 67303348e29f44beb4f914f73a4b2e37, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682326852435 2023-04-24 09:00:56,953 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 532e06f2e7094989b4354fbc5740e4b8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682326852435 2023-04-24 09:00:56,953 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting bc48756af6ab4ff787d18f64d9786ab0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682326854569 2023-04-24 09:00:56,953 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting eae114232f1148978c6fc2675f53773f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682326854569 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:56,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:56,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:56,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:56,955 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:56,955 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:56,955 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:56,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:56,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:56,958 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#B#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:56,960 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#A#compaction#460 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:57,055 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:57,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:57,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:57,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:57,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:57,058 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:57,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:57,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:57,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:57,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:57,058 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:57,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:57,063 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:57,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:57,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:57,063 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:57,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:57,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:57,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:57,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:57,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:57,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:57,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:57,064 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:57,064 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:57,064 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:57,064 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 09:00:57,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:57,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:57,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:57,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:57,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:57,065 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:57,073 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042444ddf223875349f0a8f318a7218a7618 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042444ddf223875349f0a8f318a7218a7618 2023-04-24 09:00:57,377 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/920ab053d3284e9498adcd584725ee1f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/920ab053d3284e9498adcd584725ee1f 2023-04-24 09:00:57,381 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/B of d6e3ad0fadc5b98d862d4692cd3586da into 920ab053d3284e9498adcd584725ee1f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:57,381 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:57,381 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/B, priority=13, startTime=1682326856951; duration=0sec 2023-04-24 09:00:57,381 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:00:57,381 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-24 09:00:57,382 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-24 09:00:57,382 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/C is initiating minor compaction (all files) 2023-04-24 09:00:57,382 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/C in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:57,382 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/2df0aea543394b70af88c5d1847da837, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5aa6cb8c4a4f48c0aa530466fea198dc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/eecd9766a802452e9d78751101bfec09] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=35.3 K 2023-04-24 09:00:57,382 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 2df0aea543394b70af88c5d1847da837, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1682326851813 2023-04-24 09:00:57,383 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 5aa6cb8c4a4f48c0aa530466fea198dc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682326852435 2023-04-24 09:00:57,383 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting eecd9766a802452e9d78751101bfec09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682326854569 2023-04-24 09:00:57,387 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#C#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:00:57,389 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/9ba3c23ea91048a98c78309153ad56cf as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/9ba3c23ea91048a98c78309153ad56cf 2023-04-24 09:00:57,392 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/A of d6e3ad0fadc5b98d862d4692cd3586da into 9ba3c23ea91048a98c78309153ad56cf(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:57,392 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:57,392 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/A, priority=13, startTime=1682326856951; duration=0sec 2023-04-24 09:00:57,392 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:57,488 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/03bd5c4948af41c99571d7eb1b29d7c4 2023-04-24 09:00:57,795 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/9c97b82cb9734c909665d116147325b6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9c97b82cb9734c909665d116147325b6 2023-04-24 09:00:57,797 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/C of d6e3ad0fadc5b98d862d4692cd3586da into 9c97b82cb9734c909665d116147325b6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:00:57,797 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:57,797 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/C, priority=13, startTime=1682326856951; duration=0sec 2023-04-24 09:00:57,797 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:00:57,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:57,821 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:57,846 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326857837, max timestamp=1682326857845Num uniques 859; 2023-04-24 09:00:57,846 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:57,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326857837, max timestamp=1682326857845Num uniques 859; 2023-04-24 09:00:57,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:57,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326857837, max timestamp=1682326857845Num uniques 859; 2023-04-24 09:00:57,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:00:57,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:57,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326917865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:57,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:57,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326917866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:57,896 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/9f041073b74d4dd4ab3898a25ac3d998 2023-04-24 09:00:57,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:57,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326917969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:57,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:57,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326917970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:58,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326918173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:58,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326918173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,304 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/56b6e7d59d9d4f09a2c724bb7fa35c24 2023-04-24 09:00:58,307 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/03bd5c4948af41c99571d7eb1b29d7c4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/03bd5c4948af41c99571d7eb1b29d7c4 2023-04-24 09:00:58,309 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/03bd5c4948af41c99571d7eb1b29d7c4, entries=150, sequenceid=165, filesize=25.3 K 2023-04-24 09:00:58,310 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/9f041073b74d4dd4ab3898a25ac3d998 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/9f041073b74d4dd4ab3898a25ac3d998 2023-04-24 09:00:58,312 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/9f041073b74d4dd4ab3898a25ac3d998, entries=150, sequenceid=165, filesize=11.8 K 2023-04-24 09:00:58,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/56b6e7d59d9d4f09a2c724bb7fa35c24 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/56b6e7d59d9d4f09a2c724bb7fa35c24 2023-04-24 09:00:58,315 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/56b6e7d59d9d4f09a2c724bb7fa35c24, entries=150, sequenceid=165, filesize=11.8 K 2023-04-24 09:00:58,316 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d6e3ad0fadc5b98d862d4692cd3586da in 1252ms, sequenceid=165, compaction requested=false 2023-04-24 09:00:58,316 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:58,316 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:58,316 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:58,316 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:58,316 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,316 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:58,316 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:58,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,321 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:58,322 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,322 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:58,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:58,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@660d5da5[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:58,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:58,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,330 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,330 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:58,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:58,330 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:58,330 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:58,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:58,338 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:58,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:58,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:58,340 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:58,340 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:58,340 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:58,340 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:58,340 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:58,346 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,346 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:58,346 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:58,347 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,348 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:58,355 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:58,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:58,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:58,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,355 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:58,355 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:58,355 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,455 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:58,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:58,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:58,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:58,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:58,458 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:58,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:58,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:58,458 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:58,459 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,459 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:58,463 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:58,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:58,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,463 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:58,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:58,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:58,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:58,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:58,464 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:58,464 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:58,464 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:58,464 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:58,464 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:58,464 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:58,465 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:58,465 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-24 09:00:58,465 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:58,465 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:58,465 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:58,465 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:58,465 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:58,465 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:00:58,473 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042448746393b7c84362bfa96b69e32d6f65 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042448746393b7c84362bfa96b69e32d6f65 2023-04-24 09:00:58,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:58,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:58,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326918487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:58,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326918491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:58,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326918592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:58,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326918594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:58,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326918794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:58,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326918797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,877 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/04f96200be784da692e977acfbea2c96 2023-04-24 09:00:58,893 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/7abcc120979f4975b8f5bd6c7d7dda17 2023-04-24 09:00:58,914 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/72a850dfeff440fb879b523727eeffa9 2023-04-24 09:00:58,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/04f96200be784da692e977acfbea2c96 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/04f96200be784da692e977acfbea2c96 2023-04-24 09:00:58,922 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/04f96200be784da692e977acfbea2c96, entries=150, sequenceid=203, filesize=25.3 K 2023-04-24 09:00:58,922 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/7abcc120979f4975b8f5bd6c7d7dda17 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7abcc120979f4975b8f5bd6c7d7dda17 2023-04-24 09:00:58,929 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7abcc120979f4975b8f5bd6c7d7dda17, entries=150, sequenceid=203, filesize=11.8 K 2023-04-24 09:00:58,929 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/72a850dfeff440fb879b523727eeffa9 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/72a850dfeff440fb879b523727eeffa9 2023-04-24 09:00:58,932 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/72a850dfeff440fb879b523727eeffa9, entries=150, sequenceid=203, filesize=11.8 K 2023-04-24 09:00:58,932 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for d6e3ad0fadc5b98d862d4692cd3586da in 467ms, sequenceid=203, compaction requested=true 2023-04-24 09:00:58,932 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:58,932 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:58,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:58,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:58,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,932 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:58,933 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:58,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:58,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:58,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:58,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46b46c04[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:58,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:58,939 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,946 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,946 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:58,946 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:58,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,946 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:58,946 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:58,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:58,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:58,955 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:58,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:58,956 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:58,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:58,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:58,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:58,956 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:58,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:58,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:58,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:58,963 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,964 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:58,971 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:58,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:58,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:58,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:58,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:58,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:58,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:00:58,971 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:58,972 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:58,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:58,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:58,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,972 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:58,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:58,972 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,072 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:00:59,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:00:59,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:00:59,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:00:59,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:00:59,074 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:00:59,074 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:59,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:00:59,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:00:59,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,075 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:59,080 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:59,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:59,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,080 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:00:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:59,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:00:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:59,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:00:59,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:00:59,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:00:59,081 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:00:59,081 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:00:59,081 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:00:59,081 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:00:59,081 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:00:59,081 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:59,081 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:00:59,081 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 09:00:59,081 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:00:59,081 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:59,081 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:00:59,081 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:59,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:00:59,082 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:00:59,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:00:59,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:00:59,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326859116, max timestamp=1682326859131Num uniques 859; 2023-04-24 09:00:59,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:59,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326859116, max timestamp=1682326859131Num uniques 859; 2023-04-24 09:00:59,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:59,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682326859116, max timestamp=1682326859131Num uniques 859; 2023-04-24 09:00:59,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-24 09:00:59,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:59,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326919153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:59,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326919155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:59,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326919257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326919258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:59,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326919459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:59,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326919461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,491 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424bac07b638ffe4cbf97ab2524d570379f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bac07b638ffe4cbf97ab2524d570379f 2023-04-24 09:00:59,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326919761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:00:59,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326919766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,895 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=214, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/8b73905f762c479fb51b98dc7954c182 2023-04-24 09:00:59,902 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/c1b6cf7ab4284c92a827a56f45cab17c 2023-04-24 09:00:59,910 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/244ffe8d45e3487a8adf55c7f26ff093 2023-04-24 09:00:59,912 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/8b73905f762c479fb51b98dc7954c182 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8b73905f762c479fb51b98dc7954c182 2023-04-24 09:00:59,915 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8b73905f762c479fb51b98dc7954c182, entries=150, sequenceid=214, filesize=25.3 K 2023-04-24 09:00:59,916 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/c1b6cf7ab4284c92a827a56f45cab17c as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/c1b6cf7ab4284c92a827a56f45cab17c 2023-04-24 09:00:59,918 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/c1b6cf7ab4284c92a827a56f45cab17c, entries=150, sequenceid=214, filesize=11.8 K 2023-04-24 09:00:59,918 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/244ffe8d45e3487a8adf55c7f26ff093 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/244ffe8d45e3487a8adf55c7f26ff093 2023-04-24 09:00:59,921 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/244ffe8d45e3487a8adf55c7f26ff093, entries=150, sequenceid=214, filesize=11.8 K 2023-04-24 09:00:59,921 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d6e3ad0fadc5b98d862d4692cd3586da in 840ms, sequenceid=214, compaction requested=true 2023-04-24 09:00:59,921 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:00:59,921 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:00:59,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:00:59,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:00:59,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:00:59,921 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:00:59,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:59,930 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:59,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:59,930 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:00:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:59,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:00:59,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3facefe9[Count = 0] remaining members to acquire global barrier 2023-04-24 09:00:59,931 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:00:59,931 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:59,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:59,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:59,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:00:59,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:00:59,938 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:00:59,938 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,938 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:00:59,946 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,946 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:00:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:59,946 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:59,946 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:00:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:00:59,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:00:59,948 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:00:59,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:00:59,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:00:59,948 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:00:59,948 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:00:59,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:00:59,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,955 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:00:59,955 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:00:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:00:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,956 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:00:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:00:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:00:59,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:00:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:00:59,963 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:00:59,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:00:59,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:00:59,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:00:59,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:59,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:00:59,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:00:59,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:59,963 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:00:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:00:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:00:59,963 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:00:59,963 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:00:59,964 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:00,064 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:01:00,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:01:00,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:01:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:01:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:01:00,066 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:01:00,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:00,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:01:00,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:01:00,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:00,067 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:00,071 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:00,071 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:00,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:00,072 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:01:00,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:01:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:00,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:01:00,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:01:00,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:01:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:01:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:01:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:01:00,072 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:01:00,072 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:00,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:01:00,073 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-24 09:01:00,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:01:00,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:00,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:01:00,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:00,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:01:00,073 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:00,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:00,270 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:01:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326920285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326920285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:00,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326920389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:00,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326920389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,484 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042492d50e51c64941209b3267049b938b5e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042492d50e51c64941209b3267049b938b5e 2023-04-24 09:01:00,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:00,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326920591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:00,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326920591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,887 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=250, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/e1ccc1a8bed041b889f36a8146dcd102 2023-04-24 09:01:00,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:00,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326920894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:00,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:00,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326920896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,297 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/d915d7e0b1e5448395fde71d823fe7b0 2023-04-24 09:01:01,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:01,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326921398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326921400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,704 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/9badc4af24cf4fd886f1f7a71f5412ea 2023-04-24 09:01:01,707 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/e1ccc1a8bed041b889f36a8146dcd102 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/e1ccc1a8bed041b889f36a8146dcd102 2023-04-24 09:01:01,711 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/e1ccc1a8bed041b889f36a8146dcd102, entries=150, sequenceid=250, filesize=25.3 K 2023-04-24 09:01:01,711 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/d915d7e0b1e5448395fde71d823fe7b0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d915d7e0b1e5448395fde71d823fe7b0 2023-04-24 09:01:01,714 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d915d7e0b1e5448395fde71d823fe7b0, entries=150, sequenceid=250, filesize=11.8 K 2023-04-24 09:01:01,715 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/9badc4af24cf4fd886f1f7a71f5412ea as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9badc4af24cf4fd886f1f7a71f5412ea 2023-04-24 09:01:01,718 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9badc4af24cf4fd886f1f7a71f5412ea, entries=150, sequenceid=250, filesize=11.8 K 2023-04-24 09:01:01,719 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for d6e3ad0fadc5b98d862d4692cd3586da in 1646ms, sequenceid=250, compaction requested=true 2023-04-24 09:01:01,719 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:01,719 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:01,719 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:01:01,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:01:01,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:01:01,720 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:01:01,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:01,747 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:01,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:01,747 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:01:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:01,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:01:01,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c2ca97a[Count = 0] remaining members to acquire global barrier 2023-04-24 09:01:01,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:01:01,749 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:01,761 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:01,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:01,761 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:01:01,761 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:01:01,761 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:01:01,761 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,761 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:01:01,771 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:01:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:01,771 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:01,772 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:01:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:01,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:01:01,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:01:01,774 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:01:01,774 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:01:01,774 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:01:01,774 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:01:01,774 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:01:01,779 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,780 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:01,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,781 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,782 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:01,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:01,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:01:01,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:01:01,788 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,789 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:01:01,788 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:01,789 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:01:01,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:01,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:01,789 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:01:01,790 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:01,790 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:01,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:01,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:01,889 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:01:01,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:01:01,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:01:01,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:01:01,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:01:01,892 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:01:01,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:01,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:01:01,892 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:01:01,893 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,893 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:01,902 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:01,902 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:01,903 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:01,903 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:01:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:01,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:01:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:01,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:01:01,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:01:01,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:01:01,904 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:01:01,904 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:01,904 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:01:01,904 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:01:01,904 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:01,904 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:01:01,904 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:01:01,904 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-24 09:01:01,904 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:01:01,904 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:01,904 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:01:01,904 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:01,904 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:01:01,904 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:02,315 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424fc6ea0f54ad149a89d4ace5451f8ccec to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fc6ea0f54ad149a89d4ace5451f8ccec 2023-04-24 09:01:02,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:02,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:01:02,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:02,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326922431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:02,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:02,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326922432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:02,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:02,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326922534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:02,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:02,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326922535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:02,720 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/445156ae87c546f28f688bec8cfd6f12 2023-04-24 09:01:02,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:02,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326922737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:02,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:02,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326922738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:03,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326923040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:03,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326923042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,127 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/552248e668074827a78c30aef6ad9042 2023-04-24 09:01:03,535 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/37e141db34134eb69d0aaac115388799 2023-04-24 09:01:03,538 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/445156ae87c546f28f688bec8cfd6f12 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/445156ae87c546f28f688bec8cfd6f12 2023-04-24 09:01:03,540 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/445156ae87c546f28f688bec8cfd6f12, entries=150, sequenceid=262, filesize=25.4 K 2023-04-24 09:01:03,541 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/552248e668074827a78c30aef6ad9042 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/552248e668074827a78c30aef6ad9042 2023-04-24 09:01:03,543 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/552248e668074827a78c30aef6ad9042, entries=150, sequenceid=262, filesize=11.9 K 2023-04-24 09:01:03,544 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/37e141db34134eb69d0aaac115388799 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/37e141db34134eb69d0aaac115388799 2023-04-24 09:01:03,546 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/37e141db34134eb69d0aaac115388799, entries=150, sequenceid=262, filesize=11.9 K 2023-04-24 09:01:03,547 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d6e3ad0fadc5b98d862d4692cd3586da in 1643ms, sequenceid=262, compaction requested=true 2023-04-24 09:01:03,547 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:03,547 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:03,547 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:01:03,547 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:01:03,547 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,547 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:01:03,547 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:01:03,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:03,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-04-24 09:01:03,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:01:03,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:03,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:01:03,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:03,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:01:03,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:03,554 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,555 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,555 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,555 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:01:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:01:03,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c0352f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:01:03,556 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:01:03,556 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,559 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424d187929e036941e387e1bb8ae7917cf3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d187929e036941e387e1bb8ae7917cf3 2023-04-24 09:01:03,563 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:01:03,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:01:03,563 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:01:03,563 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,563 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:01:03,564 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=284, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/af7f877102b1466bb89b397d58d85f08 2023-04-24 09:01:03,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:03,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326923562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:03,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326923565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,571 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:01:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:03,571 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:01:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:01:03,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:01:03,573 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:01:03,573 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:01:03,573 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:01:03,573 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:01:03,573 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:01:03,579 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,579 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,579 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:03,580 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,581 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:03,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:01:03,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:01:03,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,588 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:03,588 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:01:03,588 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:03,588 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:01:03,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:03,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:03,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326923667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:03,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326923669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,688 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:01:03,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:01:03,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:01:03,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:01:03,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:01:03,691 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:01:03,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:03,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:01:03,691 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:01:03,692 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,692 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,696 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:03,696 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:03,696 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,697 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:01:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:01:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:01:03,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:01:03,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:01:03,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:01:03,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:03,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:01:03,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:01:03,697 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:01:03,697 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:03,697 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:01:03,698 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:01:03,698 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:03,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:01:03,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:01:03,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:01:03,698 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:01:03,704 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,704 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,705 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:01:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:01:03,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f1a57f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:01:03,706 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:01:03,706 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,713 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:01:03,713 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,713 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:01:03,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:01:03,713 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:01:03,721 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:01:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:03,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:01:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:01:03,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:01:03,723 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:01:03,723 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:01:03,723 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:01:03,723 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:01:03,723 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:01:03,729 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,729 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,729 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,730 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,731 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:03,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:01:03,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:01:03,738 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:03,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,738 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,738 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:01:03,738 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:01:03,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:01:03,838 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:01:03,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:01:03,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:01:03,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:01:03,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:01:03,841 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:01:03,841 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:03,841 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:01:03,841 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:01:03,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,842 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,846 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:03,846 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:03,846 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,847 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:01:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:01:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:01:03,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:01:03,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:01:03,848 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:01:03,848 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:03,848 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:01:03,852 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:01:03,852 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:03,852 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:01:03,852 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:01:03,852 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:01:03,852 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:03,852 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:01:03,852 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:01:03,852 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,852 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:01:03,852 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:01:03,863 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,863 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:01:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:01:03,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dfac58f[Count = 0] remaining members to acquire global barrier 2023-04-24 09:01:03,865 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:01:03,865 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,871 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,871 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:01:03,871 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:01:03,871 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,871 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:01:03,871 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:01:03,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:03,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326923871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:03,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326923872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,898 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:01:03,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:03,898 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:01:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:01:03,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:01:03,901 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:01:03,901 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:01:03,901 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:01:03,901 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:01:03,901 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:01:03,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,913 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:03,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:03,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:03,914 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:03,916 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:03,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:03,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:03,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:01:03,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:01:03,921 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:03,921 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:03,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:03,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:03,922 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:01:03,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:03,922 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:01:03,922 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:03,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:01:03,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/018f1710b34b4b949ac6c1b7fe9c83fb 2023-04-24 09:01:03,981 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/af6178a4ffe74dec8791794cd441ccd6 2023-04-24 09:01:03,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/af7f877102b1466bb89b397d58d85f08 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/af7f877102b1466bb89b397d58d85f08 2023-04-24 09:01:03,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/af7f877102b1466bb89b397d58d85f08, entries=250, sequenceid=284, filesize=39.2 K 2023-04-24 09:01:03,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/018f1710b34b4b949ac6c1b7fe9c83fb as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/018f1710b34b4b949ac6c1b7fe9c83fb 2023-04-24 09:01:03,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/018f1710b34b4b949ac6c1b7fe9c83fb, entries=150, sequenceid=284, filesize=11.9 K 2023-04-24 09:01:03,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/af6178a4ffe74dec8791794cd441ccd6 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/af6178a4ffe74dec8791794cd441ccd6 2023-04-24 09:01:03,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/af6178a4ffe74dec8791794cd441ccd6, entries=150, sequenceid=284, filesize=11.9 K 2023-04-24 09:01:03,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=46.96 KB/48090 for d6e3ad0fadc5b98d862d4692cd3586da in 445ms, sequenceid=284, compaction requested=true 2023-04-24 09:01:03,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:03,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:01:03,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:01:03,994 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-24 09:01:03,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:01:03,994 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-24 09:01:03,996 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 195948 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-24 09:01:03,996 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84844 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-24 09:01:03,996 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/B is initiating minor compaction (all files) 2023-04-24 09:01:03,996 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/B in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:03,996 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/920ab053d3284e9498adcd584725ee1f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/9f041073b74d4dd4ab3898a25ac3d998, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7abcc120979f4975b8f5bd6c7d7dda17, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/c1b6cf7ab4284c92a827a56f45cab17c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d915d7e0b1e5448395fde71d823fe7b0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/552248e668074827a78c30aef6ad9042, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/018f1710b34b4b949ac6c1b7fe9c83fb] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=82.9 K 2023-04-24 09:01:03,996 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/A is initiating minor compaction (all files) 2023-04-24 09:01:03,996 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/A in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:03,996 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/9ba3c23ea91048a98c78309153ad56cf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/03bd5c4948af41c99571d7eb1b29d7c4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/04f96200be784da692e977acfbea2c96, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8b73905f762c479fb51b98dc7954c182, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/e1ccc1a8bed041b889f36a8146dcd102, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/445156ae87c546f28f688bec8cfd6f12, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/af7f877102b1466bb89b397d58d85f08] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=191.4 K 2023-04-24 09:01:03,996 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 920ab053d3284e9498adcd584725ee1f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682326854569 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 9ba3c23ea91048a98c78309153ad56cf, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682326854569 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 9f041073b74d4dd4ab3898a25ac3d998, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682326855691 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 03bd5c4948af41c99571d7eb1b29d7c4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682326855691 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 7abcc120979f4975b8f5bd6c7d7dda17, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682326857861 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 04f96200be784da692e977acfbea2c96, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682326857861 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting c1b6cf7ab4284c92a827a56f45cab17c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682326858481 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 8b73905f762c479fb51b98dc7954c182, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682326858481 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d915d7e0b1e5448395fde71d823fe7b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682326859138 2023-04-24 09:01:03,997 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting e1ccc1a8bed041b889f36a8146dcd102, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682326859138 2023-04-24 09:01:03,998 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 552248e668074827a78c30aef6ad9042, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682326860277 2023-04-24 09:01:03,998 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 445156ae87c546f28f688bec8cfd6f12, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682326860277 2023-04-24 09:01:03,998 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 018f1710b34b4b949ac6c1b7fe9c83fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682326862430 2023-04-24 09:01:03,998 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting af7f877102b1466bb89b397d58d85f08, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682326862425 2023-04-24 09:01:04,006 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#B#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:01:04,009 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#A#compaction#480 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:01:04,013 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/a7208c78d25e4d0ab045230427c048a2 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/a7208c78d25e4d0ab045230427c048a2 2023-04-24 09:01:04,016 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/B of d6e3ad0fadc5b98d862d4692cd3586da into a7208c78d25e4d0ab045230427c048a2(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:01:04,016 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:04,016 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/B, priority=9, startTime=1682326863994; duration=0sec 2023-04-24 09:01:04,016 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:01:04,017 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-24 09:01:04,018 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84844 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-24 09:01:04,018 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/C is initiating minor compaction (all files) 2023-04-24 09:01:04,018 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/C in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:04,018 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9c97b82cb9734c909665d116147325b6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/56b6e7d59d9d4f09a2c724bb7fa35c24, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/72a850dfeff440fb879b523727eeffa9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/244ffe8d45e3487a8adf55c7f26ff093, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9badc4af24cf4fd886f1f7a71f5412ea, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/37e141db34134eb69d0aaac115388799, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/af6178a4ffe74dec8791794cd441ccd6] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=82.9 K 2023-04-24 09:01:04,019 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 9c97b82cb9734c909665d116147325b6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1682326854569 2023-04-24 09:01:04,019 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 56b6e7d59d9d4f09a2c724bb7fa35c24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682326855691 2023-04-24 09:01:04,019 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 72a850dfeff440fb879b523727eeffa9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1682326857861 2023-04-24 09:01:04,019 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 244ffe8d45e3487a8adf55c7f26ff093, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1682326858481 2023-04-24 09:01:04,020 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 9badc4af24cf4fd886f1f7a71f5412ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682326859138 2023-04-24 09:01:04,020 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 37e141db34134eb69d0aaac115388799, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1682326860277 2023-04-24 09:01:04,020 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting af6178a4ffe74dec8791794cd441ccd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682326862430 2023-04-24 09:01:04,022 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:01:04,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:01:04,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:01:04,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:01:04,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:01:04,025 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:01:04,025 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:04,025 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:01:04,025 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:01:04,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:04,026 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:04,028 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#C#compaction#482 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:01:04,036 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:04,036 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:04,036 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,036 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:01:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:04,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:01:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:04,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:01:04,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:01:04,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:01:04,040 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:01:04,040 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:04,040 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:01:04,040 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:01:04,040 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:01:04,040 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:04,040 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:01:04,040 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-24 09:01:04,040 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:01:04,040 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:04,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:01:04,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:04,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:01:04,041 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:04,043 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/a8fa208494bf4b638f26787e8c4f2fc1 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/a8fa208494bf4b638f26787e8c4f2fc1 2023-04-24 09:01:04,060 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/C of d6e3ad0fadc5b98d862d4692cd3586da into a8fa208494bf4b638f26787e8c4f2fc1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:01:04,060 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:04,060 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/C, priority=9, startTime=1682326863994; duration=0sec 2023-04-24 09:01:04,060 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:01:04,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:04,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:01:04,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682326864196, max timestamp=1682326864216Num uniques 859; 2023-04-24 09:01:04,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:01:04,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326864196, max timestamp=1682326864216Num uniques 859; 2023-04-24 09:01:04,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:01:04,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682326864196, max timestamp=1682326864216Num uniques 859; 2023-04-24 09:01:04,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase16:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-24 09:01:04,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326924241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326924243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326924346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326924347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,465 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042421063f95286d4b21a76bb5a2107c13b6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042421063f95286d4b21a76bb5a2107c13b6 2023-04-24 09:01:04,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326924549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326924550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,822 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/011816dfd2bb480e8159d0bb8beb3aa0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/011816dfd2bb480e8159d0bb8beb3aa0 2023-04-24 09:01:04,824 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/A of d6e3ad0fadc5b98d862d4692cd3586da into 011816dfd2bb480e8159d0bb8beb3aa0(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:01:04,824 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:04,825 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/A, priority=9, startTime=1682326863994; duration=0sec 2023-04-24 09:01:04,825 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:01:04,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326924854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326924855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,874 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=295, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/62520ec8f2af4e0a86f5ae93ab77ecb4 2023-04-24 09:01:04,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:04,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34848 deadline: 1682326924952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:04,955 DEBUG [Thread-3270] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:01:05,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:05,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34896 deadline: 1682326925001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,004 DEBUG [Thread-3272] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:01:05,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:05,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 188.40.62.62:34876 deadline: 1682326925023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,025 DEBUG [Thread-3268] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18214 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., hostname=jenkins-hbase16.apache.org,42837,1682326693550, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-24 09:01:05,281 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/f1ecd2efc19142838cf4df75b1c5cb56 2023-04-24 09:01:05,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:05,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326925358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:05,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326925361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,690 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/3a527a664518437a9346d0bc6fb7c8de 2023-04-24 09:01:05,693 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/62520ec8f2af4e0a86f5ae93ab77ecb4 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/62520ec8f2af4e0a86f5ae93ab77ecb4 2023-04-24 09:01:05,696 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/62520ec8f2af4e0a86f5ae93ab77ecb4, entries=150, sequenceid=295, filesize=25.4 K 2023-04-24 09:01:05,696 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/f1ecd2efc19142838cf4df75b1c5cb56 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/f1ecd2efc19142838cf4df75b1c5cb56 2023-04-24 09:01:05,699 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/f1ecd2efc19142838cf4df75b1c5cb56, entries=150, sequenceid=295, filesize=11.9 K 2023-04-24 09:01:05,699 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/3a527a664518437a9346d0bc6fb7c8de as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3a527a664518437a9346d0bc6fb7c8de 2023-04-24 09:01:05,702 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3a527a664518437a9346d0bc6fb7c8de, entries=150, sequenceid=295, filesize=11.9 K 2023-04-24 09:01:05,702 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d6e3ad0fadc5b98d862d4692cd3586da in 1662ms, sequenceid=295, compaction requested=false 2023-04-24 09:01:05,702 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:05,702 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:05,702 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:01:05,702 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:01:05,703 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,703 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:01:05,703 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:01:05,721 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:05,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:05,721 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:01:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:05,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:01:05,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54b4d433[Count = 0] remaining members to acquire global barrier 2023-04-24 09:01:05,723 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:01:05,723 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:05,779 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:05,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:05,780 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,780 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:01:05,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:01:05,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:01:05,780 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:01:05,877 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,877 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:01:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:05,878 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:05,878 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:01:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:01:05,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:01:05,880 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:01:05,880 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:01:05,880 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:01:05,880 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:01:05,880 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:01:05,929 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:05,929 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:05,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:05,929 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:05,929 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:05,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:05,930 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,938 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:05,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:01:05,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:05,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:05,938 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:01:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:05,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:05,938 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:05,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:05,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:05,938 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:06,038 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:01:06,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:01:06,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(933): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-04-24 09:01:06,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-24 09:01:06,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-24 09:01:06,041 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-24 09:01:06,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:06,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-24 09:01:06,041 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-24 09:01:06,042 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:06,042 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:06,046 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:06,046 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:06,046 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:06,046 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-24 09:01:06,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-24 09:01:06,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:06,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-24 09:01:06,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-24 09:01:06,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-24 09:01:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-24 09:01:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-24 09:01:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-24 09:01:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-24 09:01:06,047 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-24 09:01:06,047 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:06,047 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. started... 2023-04-24 09:01:06,048 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-24 09:01:06,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:01:06,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:06,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:01:06,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:06,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:01:06,048 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-24 09:01:06,057 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424ff7f82bdddd4416bab4fff10ac004fa0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ff7f82bdddd4416bab4fff10ac004fa0 2023-04-24 09:01:06,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:06,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. as already flushing 2023-04-24 09:01:06,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:06,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326926379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:06,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326926380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,464 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=333, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/7f3fd6b398cf488e8e0b01424dbba6d7 2023-04-24 09:01:06,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:06,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326926481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:06,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326926483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:06,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42837] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326926684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:06,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42837] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326926687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,751 DEBUG [Thread-3285] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a82cded to 127.0.0.1:63978 2023-04-24 09:01:06,751 DEBUG [Thread-3285] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:06,752 DEBUG [Thread-3277] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27489815 to 127.0.0.1:63978 2023-04-24 09:01:06,752 DEBUG [Thread-3283] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17856520 to 127.0.0.1:63978 2023-04-24 09:01:06,752 DEBUG [Thread-3277] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:06,752 DEBUG [Thread-3283] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:06,753 DEBUG [Thread-3279] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33765ee2 to 127.0.0.1:63978 2023-04-24 09:01:06,753 DEBUG [Thread-3279] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:06,754 DEBUG [Thread-3281] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16e1a968 to 127.0.0.1:63978 2023-04-24 09:01:06,754 DEBUG [Thread-3281] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:06,871 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/d508f9c69747403b9bb7134ce64d8c37 2023-04-24 09:01:06,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:06,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34858 deadline: 1682326926987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:06,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-24 09:01:06,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 188.40.62.62:34874 deadline: 1682326926992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,283 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=333 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/e6c41d0ff9cb44c6a9501771fc212402 2023-04-24 09:01:07,293 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/7f3fd6b398cf488e8e0b01424dbba6d7 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7f3fd6b398cf488e8e0b01424dbba6d7 2023-04-24 09:01:07,299 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7f3fd6b398cf488e8e0b01424dbba6d7, entries=150, sequenceid=333, filesize=25.4 K 2023-04-24 09:01:07,300 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/d508f9c69747403b9bb7134ce64d8c37 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d508f9c69747403b9bb7134ce64d8c37 2023-04-24 09:01:07,305 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d508f9c69747403b9bb7134ce64d8c37, entries=150, sequenceid=333, filesize=11.9 K 2023-04-24 09:01:07,307 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/e6c41d0ff9cb44c6a9501771fc212402 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/e6c41d0ff9cb44c6a9501771fc212402 2023-04-24 09:01:07,312 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/e6c41d0ff9cb44c6a9501771fc212402, entries=150, sequenceid=333, filesize=11.9 K 2023-04-24 09:01:07,313 INFO [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for d6e3ad0fadc5b98d862d4692cd3586da in 1265ms, sequenceid=333, compaction requested=true 2023-04-24 09:01:07,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:07,313 DEBUG [rs(jenkins-hbase16.apache.org,42837,1682326693550)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:07,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-24 09:01:07,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-24 09:01:07,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-24 09:01:07,313 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-24 09:01:07,321 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:07,321 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:07,321 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:07,321 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-24 09:01:07,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:07,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:07,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:07,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:07,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,42837,1682326693550' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-24 09:01:07,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159cfac7[Count = 0] remaining members to acquire global barrier 2023-04-24 09:01:07,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-24 09:01:07,323 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:07,329 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:07,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:07,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:07,330 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-24 09:01:07,330 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-24 09:01:07,330 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase16.apache.org,42837,1682326693550' in zk 2023-04-24 09:01:07,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,330 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-24 09:01:07,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-24 09:01:07,338 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:07,338 DEBUG [member: 'jenkins-hbase16.apache.org,42837,1682326693550' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-24 09:01:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:07,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:07,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:07,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase16.apache.org,42837,1682326693550': 2023-04-24 09:01:07,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,42837,1682326693550' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-24 09:01:07,342 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-24 09:01:07,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-24 09:01:07,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-24 09:01:07,343 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-24 09:01:07,343 INFO [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-24 09:01:07,352 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:07,352 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:07,353 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-24 09:01:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-24 09:01:07,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:07,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:07,354 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-24 09:01:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-24 09:01:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:07,356 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-24 09:01:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-24 09:01:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-24 09:01:07,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-24 09:01:07,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-24 09:01:07,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:07,369 DEBUG [(jenkins-hbase16.apache.org,40429,1682326692433)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-24 09:01:07,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-24 09:01:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-24 09:01:07,369 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-24 09:01:07,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-24 09:01:07,370 DEBUG [Thread-3276] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-24 09:01:07,371 DEBUG [Thread-3276] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-24 09:01:07,370 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-24 09:01:07,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-24 09:01:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-24 09:01:07,371 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:07,371 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:07,371 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-24 09:01:07,371 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-24 09:01:07,471 DEBUG [Thread-3276] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-24 09:01:07,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-24 09:01:07,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42837] regionserver.HRegion(9158): Flush requested on d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:07,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-24 09:01:07,495 DEBUG [Thread-3274] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ef34de3 to 127.0.0.1:63978 2023-04-24 09:01:07,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:01:07,496 DEBUG [Thread-3274] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:07,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:07,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:01:07,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:07,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:01:07,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:07,501 DEBUG [Thread-3266] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x768816e9 to 127.0.0.1:63978 2023-04-24 09:01:07,501 DEBUG [Thread-3266] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:07,919 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230424f9acca6720fe456d993fc12fe9e7c5c3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f9acca6720fe456d993fc12fe9e7c5c3 2023-04-24 09:01:08,326 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=344, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/5b2f014c78eb49a38edabd582ff9bb33 2023-04-24 09:01:08,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/2da7c4bd6e6b401b86de0137d5197b58 2023-04-24 09:01:09,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=344 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/0d53ced2cbe54e138eae68c41467296f 2023-04-24 09:01:09,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/5b2f014c78eb49a38edabd582ff9bb33 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/5b2f014c78eb49a38edabd582ff9bb33 2023-04-24 09:01:09,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/5b2f014c78eb49a38edabd582ff9bb33, entries=150, sequenceid=344, filesize=25.4 K 2023-04-24 09:01:09,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/2da7c4bd6e6b401b86de0137d5197b58 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/2da7c4bd6e6b401b86de0137d5197b58 2023-04-24 09:01:09,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/2da7c4bd6e6b401b86de0137d5197b58, entries=150, sequenceid=344, filesize=11.9 K 2023-04-24 09:01:09,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/0d53ced2cbe54e138eae68c41467296f as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0d53ced2cbe54e138eae68c41467296f 2023-04-24 09:01:09,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0d53ced2cbe54e138eae68c41467296f, entries=150, sequenceid=344, filesize=11.9 K 2023-04-24 09:01:09,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=6.71 KB/6870 for d6e3ad0fadc5b98d862d4692cd3586da in 1679ms, sequenceid=344, compaction requested=true 2023-04-24 09:01:09,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:09,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:01:09,174 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:01:09,174 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:01:09,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:01:09,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:01:09,175 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104764 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:01:09,175 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:01:09,175 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/A is initiating minor compaction (all files) 2023-04-24 09:01:09,175 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/B is initiating minor compaction (all files) 2023-04-24 09:01:09,175 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/A in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:09,175 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/B in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:09,175 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/011816dfd2bb480e8159d0bb8beb3aa0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/62520ec8f2af4e0a86f5ae93ab77ecb4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7f3fd6b398cf488e8e0b01424dbba6d7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/5b2f014c78eb49a38edabd582ff9bb33] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=102.3 K 2023-04-24 09:01:09,175 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/a7208c78d25e4d0ab045230427c048a2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/f1ecd2efc19142838cf4df75b1c5cb56, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d508f9c69747403b9bb7134ce64d8c37, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/2da7c4bd6e6b401b86de0137d5197b58] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=48.2 K 2023-04-24 09:01:09,175 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 011816dfd2bb480e8159d0bb8beb3aa0, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682326862430 2023-04-24 09:01:09,175 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a7208c78d25e4d0ab045230427c048a2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682326862430 2023-04-24 09:01:09,176 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 62520ec8f2af4e0a86f5ae93ab77ecb4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682326863557 2023-04-24 09:01:09,176 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting f1ecd2efc19142838cf4df75b1c5cb56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682326863557 2023-04-24 09:01:09,176 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 7f3fd6b398cf488e8e0b01424dbba6d7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682326864230 2023-04-24 09:01:09,176 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting d508f9c69747403b9bb7134ce64d8c37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682326864230 2023-04-24 09:01:09,176 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] compactions.Compactor(207): Compacting 5b2f014c78eb49a38edabd582ff9bb33, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682326866377 2023-04-24 09:01:09,176 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 2da7c4bd6e6b401b86de0137d5197b58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682326866377 2023-04-24 09:01:09,184 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#B#compaction#493 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:01:09,184 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#A#compaction#492 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:01:09,598 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/39ba5f704e4d4caab596e02363374750 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/39ba5f704e4d4caab596e02363374750 2023-04-24 09:01:09,602 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/B of d6e3ad0fadc5b98d862d4692cd3586da into 39ba5f704e4d4caab596e02363374750(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:01:09,602 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:09,602 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/B, priority=12, startTime=1682326869174; duration=0sec 2023-04-24 09:01:09,602 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-24 09:01:09,602 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-24 09:01:09,603 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-24 09:01:09,603 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1912): d6e3ad0fadc5b98d862d4692cd3586da/C is initiating minor compaction (all files) 2023-04-24 09:01:09,603 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6e3ad0fadc5b98d862d4692cd3586da/C in TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:09,603 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/a8fa208494bf4b638f26787e8c4f2fc1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3a527a664518437a9346d0bc6fb7c8de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/e6c41d0ff9cb44c6a9501771fc212402, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0d53ced2cbe54e138eae68c41467296f] into tmpdir=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp, totalSize=48.2 K 2023-04-24 09:01:09,603 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting a8fa208494bf4b638f26787e8c4f2fc1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682326862430 2023-04-24 09:01:09,603 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 3a527a664518437a9346d0bc6fb7c8de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1682326863557 2023-04-24 09:01:09,603 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting e6c41d0ff9cb44c6a9501771fc212402, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=333, earliestPutTs=1682326864230 2023-04-24 09:01:09,604 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] compactions.Compactor(207): Compacting 0d53ced2cbe54e138eae68c41467296f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=344, earliestPutTs=1682326866377 2023-04-24 09:01:09,609 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] throttle.PressureAwareThroughputController(145): d6e3ad0fadc5b98d862d4692cd3586da#C#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-24 09:01:10,016 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/f29e04b16c9547cb8c5b34f7d05a84a5 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/f29e04b16c9547cb8c5b34f7d05a84a5 2023-04-24 09:01:10,020 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/C of d6e3ad0fadc5b98d862d4692cd3586da into f29e04b16c9547cb8c5b34f7d05a84a5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-24 09:01:10,020 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:10,020 INFO [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/C, priority=12, startTime=1682326869174; duration=0sec 2023-04-24 09:01:10,020 DEBUG [RS:0;jenkins-hbase16:42837-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:01:10,414 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/7a3a58b6492a416080e5021dda85b5bd as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7a3a58b6492a416080e5021dda85b5bd 2023-04-24 09:01:10,419 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d6e3ad0fadc5b98d862d4692cd3586da/A of d6e3ad0fadc5b98d862d4692cd3586da into 7a3a58b6492a416080e5021dda85b5bd(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-24 09:01:10,419 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:10,419 INFO [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da., storeName=d6e3ad0fadc5b98d862d4692cd3586da/A, priority=12, startTime=1682326869174; duration=1sec 2023-04-24 09:01:10,419 DEBUG [RS:0;jenkins-hbase16:42837-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-24 09:01:15,017 DEBUG [Thread-3270] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00fcbd84 to 127.0.0.1:63978 2023-04-24 09:01:15,018 DEBUG [Thread-3270] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:15,083 DEBUG [Thread-3272] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18884189 to 127.0.0.1:63978 2023-04-24 09:01:15,083 DEBUG [Thread-3272] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:15,120 DEBUG [Thread-3268] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27acfde4 to 127.0.0.1:63978 2023-04-24 09:01:15,121 DEBUG [Thread-3268] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:15,121 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-24 09:01:15,121 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-04-24 09:01:15,121 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-24 09:01:15,121 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-24 09:01:15,121 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 9 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 6347 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5898 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5846 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 6340 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(398): read 5867 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-24 09:01:15,122 INFO [Listener at localhost.localdomain/44291] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 09:01:15,122 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x085873d0 to 127.0.0.1:63978 2023-04-24 09:01:15,122 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:15,123 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-24 09:01:15,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable TestAcidGuarantees 2023-04-24 09:01:15,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-24 09:01:15,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 09:01:15,130 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326875130"}]},"ts":"1682326875130"} 2023-04-24 09:01:15,131 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-24 09:01:15,146 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-24 09:01:15,147 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, UNASSIGN}] 2023-04-24 09:01:15,148 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, UNASSIGN 2023-04-24 09:01:15,149 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=d6e3ad0fadc5b98d862d4692cd3586da, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:15,149 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326875149"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1682326875149"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682326875149"}]},"ts":"1682326875149"} 2023-04-24 09:01:15,151 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550}] 2023-04-24 09:01:15,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 09:01:15,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(109): Close d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:15,303 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing d6e3ad0fadc5b98d862d4692cd3586da, disabling compactions & flushes 2023-04-24 09:01:15,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:15,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:15,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. after waiting 0 ms 2023-04-24 09:01:15,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:15,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing d6e3ad0fadc5b98d862d4692cd3586da 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-24 09:01:15,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=A 2023-04-24 09:01:15,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:15,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=B 2023-04-24 09:01:15,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:15,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d6e3ad0fadc5b98d862d4692cd3586da, store=C 2023-04-24 09:01:15,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-24 09:01:15,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 09:01:15,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304245c46d3ee93fa4d08950a17691b395c1a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245c46d3ee93fa4d08950a17691b395c1a 2023-04-24 09:01:15,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 09:01:16,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=354, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/ec3b0cae57da4816ae42a80bc05bb44a 2023-04-24 09:01:16,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 09:01:16,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/730c26ee91a445338c466f2f95c6745b 2023-04-24 09:01:16,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/3606fdc7bf3f466f8a0e009dfacd8475 2023-04-24 09:01:16,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/A/ec3b0cae57da4816ae42a80bc05bb44a as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ec3b0cae57da4816ae42a80bc05bb44a 2023-04-24 09:01:16,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ec3b0cae57da4816ae42a80bc05bb44a, entries=100, sequenceid=354, filesize=18.6 K 2023-04-24 09:01:16,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/B/730c26ee91a445338c466f2f95c6745b as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/730c26ee91a445338c466f2f95c6745b 2023-04-24 09:01:16,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/730c26ee91a445338c466f2f95c6745b, entries=100, sequenceid=354, filesize=9.5 K 2023-04-24 09:01:16,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/.tmp/C/3606fdc7bf3f466f8a0e009dfacd8475 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3606fdc7bf3f466f8a0e009dfacd8475 2023-04-24 09:01:16,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3606fdc7bf3f466f8a0e009dfacd8475, entries=100, sequenceid=354, filesize=9.5 K 2023-04-24 09:01:16,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d6e3ad0fadc5b98d862d4692cd3586da in 1679ms, sequenceid=354, compaction requested=false 2023-04-24 09:01:17,004 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ed2d9c2fdfbe49c6b08f325243877deb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/01e907ae82bf4d27af9aa640948106b7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/a790c0c09f6a4c309f80a3463a56fa7f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/32914c9710a64b139343f387b831620f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8854b96e331646a595077c8af3f083b3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/852d85939025427bb7e0d3d80c7d7d46, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/f1fcfe1c8c5c4ab089b5955896279550, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/b2871a2964fc4a5b977f3ed3c1628b0b, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/67303348e29f44beb4f914f73a4b2e37, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/9ba3c23ea91048a98c78309153ad56cf, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/bc48756af6ab4ff787d18f64d9786ab0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/03bd5c4948af41c99571d7eb1b29d7c4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/04f96200be784da692e977acfbea2c96, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8b73905f762c479fb51b98dc7954c182, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/e1ccc1a8bed041b889f36a8146dcd102, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/445156ae87c546f28f688bec8cfd6f12, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/af7f877102b1466bb89b397d58d85f08, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/011816dfd2bb480e8159d0bb8beb3aa0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/62520ec8f2af4e0a86f5ae93ab77ecb4, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7f3fd6b398cf488e8e0b01424dbba6d7, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/5b2f014c78eb49a38edabd582ff9bb33] to archive 2023-04-24 09:01:17,005 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:01:17,006 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ed2d9c2fdfbe49c6b08f325243877deb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ed2d9c2fdfbe49c6b08f325243877deb 2023-04-24 09:01:17,007 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/01e907ae82bf4d27af9aa640948106b7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/01e907ae82bf4d27af9aa640948106b7 2023-04-24 09:01:17,008 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/a790c0c09f6a4c309f80a3463a56fa7f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/a790c0c09f6a4c309f80a3463a56fa7f 2023-04-24 09:01:17,009 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/32914c9710a64b139343f387b831620f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/32914c9710a64b139343f387b831620f 2023-04-24 09:01:17,010 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8854b96e331646a595077c8af3f083b3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8854b96e331646a595077c8af3f083b3 2023-04-24 09:01:17,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/852d85939025427bb7e0d3d80c7d7d46 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/852d85939025427bb7e0d3d80c7d7d46 2023-04-24 09:01:17,011 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/f1fcfe1c8c5c4ab089b5955896279550 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/f1fcfe1c8c5c4ab089b5955896279550 2023-04-24 09:01:17,012 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/b2871a2964fc4a5b977f3ed3c1628b0b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/b2871a2964fc4a5b977f3ed3c1628b0b 2023-04-24 09:01:17,013 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/67303348e29f44beb4f914f73a4b2e37 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/67303348e29f44beb4f914f73a4b2e37 2023-04-24 09:01:17,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/9ba3c23ea91048a98c78309153ad56cf to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/9ba3c23ea91048a98c78309153ad56cf 2023-04-24 09:01:17,014 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/bc48756af6ab4ff787d18f64d9786ab0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/bc48756af6ab4ff787d18f64d9786ab0 2023-04-24 09:01:17,015 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/03bd5c4948af41c99571d7eb1b29d7c4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/03bd5c4948af41c99571d7eb1b29d7c4 2023-04-24 09:01:17,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/04f96200be784da692e977acfbea2c96 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/04f96200be784da692e977acfbea2c96 2023-04-24 09:01:17,016 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8b73905f762c479fb51b98dc7954c182 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/8b73905f762c479fb51b98dc7954c182 2023-04-24 09:01:17,017 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/e1ccc1a8bed041b889f36a8146dcd102 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/e1ccc1a8bed041b889f36a8146dcd102 2023-04-24 09:01:17,018 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/445156ae87c546f28f688bec8cfd6f12 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/445156ae87c546f28f688bec8cfd6f12 2023-04-24 09:01:17,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/af7f877102b1466bb89b397d58d85f08 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/af7f877102b1466bb89b397d58d85f08 2023-04-24 09:01:17,019 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/011816dfd2bb480e8159d0bb8beb3aa0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/011816dfd2bb480e8159d0bb8beb3aa0 2023-04-24 09:01:17,020 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/62520ec8f2af4e0a86f5ae93ab77ecb4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/62520ec8f2af4e0a86f5ae93ab77ecb4 2023-04-24 09:01:17,021 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7f3fd6b398cf488e8e0b01424dbba6d7 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7f3fd6b398cf488e8e0b01424dbba6d7 2023-04-24 09:01:17,022 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/5b2f014c78eb49a38edabd582ff9bb33 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/5b2f014c78eb49a38edabd582ff9bb33 2023-04-24 09:01:17,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/14703ee64ec4446dac3c260802d9847e, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7ddaffa3809f4fd3bc9bbebbb34aee12, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/19a8b684a95f434b9b8dc51b369c4909, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/38c641fc75f6449a95001db498cd01ab, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/37efcf69aa7944fb8960e21d0927a370, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/76665c93fe5545528b8c0e9743389524, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5843ec43c5c940049c7c6610e453062a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5358e906f9884a6d8a3eac72be8b0cc0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/532e06f2e7094989b4354fbc5740e4b8, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/920ab053d3284e9498adcd584725ee1f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/eae114232f1148978c6fc2675f53773f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/9f041073b74d4dd4ab3898a25ac3d998, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7abcc120979f4975b8f5bd6c7d7dda17, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/c1b6cf7ab4284c92a827a56f45cab17c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d915d7e0b1e5448395fde71d823fe7b0, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/552248e668074827a78c30aef6ad9042, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/a7208c78d25e4d0ab045230427c048a2, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/018f1710b34b4b949ac6c1b7fe9c83fb, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/f1ecd2efc19142838cf4df75b1c5cb56, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d508f9c69747403b9bb7134ce64d8c37, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/2da7c4bd6e6b401b86de0137d5197b58] to archive 2023-04-24 09:01:17,024 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:01:17,025 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/14703ee64ec4446dac3c260802d9847e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/14703ee64ec4446dac3c260802d9847e 2023-04-24 09:01:17,026 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7ddaffa3809f4fd3bc9bbebbb34aee12 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7ddaffa3809f4fd3bc9bbebbb34aee12 2023-04-24 09:01:17,026 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/19a8b684a95f434b9b8dc51b369c4909 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/19a8b684a95f434b9b8dc51b369c4909 2023-04-24 09:01:17,027 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/38c641fc75f6449a95001db498cd01ab to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/38c641fc75f6449a95001db498cd01ab 2023-04-24 09:01:17,028 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/37efcf69aa7944fb8960e21d0927a370 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/37efcf69aa7944fb8960e21d0927a370 2023-04-24 09:01:17,029 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/76665c93fe5545528b8c0e9743389524 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/76665c93fe5545528b8c0e9743389524 2023-04-24 09:01:17,029 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5843ec43c5c940049c7c6610e453062a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5843ec43c5c940049c7c6610e453062a 2023-04-24 09:01:17,030 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5358e906f9884a6d8a3eac72be8b0cc0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/5358e906f9884a6d8a3eac72be8b0cc0 2023-04-24 09:01:17,031 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/532e06f2e7094989b4354fbc5740e4b8 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/532e06f2e7094989b4354fbc5740e4b8 2023-04-24 09:01:17,032 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/920ab053d3284e9498adcd584725ee1f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/920ab053d3284e9498adcd584725ee1f 2023-04-24 09:01:17,032 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/eae114232f1148978c6fc2675f53773f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/eae114232f1148978c6fc2675f53773f 2023-04-24 09:01:17,033 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/9f041073b74d4dd4ab3898a25ac3d998 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/9f041073b74d4dd4ab3898a25ac3d998 2023-04-24 09:01:17,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7abcc120979f4975b8f5bd6c7d7dda17 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/7abcc120979f4975b8f5bd6c7d7dda17 2023-04-24 09:01:17,034 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/c1b6cf7ab4284c92a827a56f45cab17c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/c1b6cf7ab4284c92a827a56f45cab17c 2023-04-24 09:01:17,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d915d7e0b1e5448395fde71d823fe7b0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d915d7e0b1e5448395fde71d823fe7b0 2023-04-24 09:01:17,035 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/552248e668074827a78c30aef6ad9042 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/552248e668074827a78c30aef6ad9042 2023-04-24 09:01:17,036 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/a7208c78d25e4d0ab045230427c048a2 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/a7208c78d25e4d0ab045230427c048a2 2023-04-24 09:01:17,037 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/018f1710b34b4b949ac6c1b7fe9c83fb to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/018f1710b34b4b949ac6c1b7fe9c83fb 2023-04-24 09:01:17,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/f1ecd2efc19142838cf4df75b1c5cb56 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/f1ecd2efc19142838cf4df75b1c5cb56 2023-04-24 09:01:17,038 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d508f9c69747403b9bb7134ce64d8c37 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/d508f9c69747403b9bb7134ce64d8c37 2023-04-24 09:01:17,039 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/2da7c4bd6e6b401b86de0137d5197b58 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/2da7c4bd6e6b401b86de0137d5197b58 2023-04-24 09:01:17,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9ee13b1bd9e54876a36bd777132ac465, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0f6015dd8d3441ca881c36fdfbc4fb0f, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/7535703bd9c14189be2b0fa353a4c1ad, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/04de6d7ee32247ffa1fdb48b61fffd4a, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5f0b42148f064de49876dafee8697a62, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/62be3ce3a7674888830178f5db28b47c, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/2df0aea543394b70af88c5d1847da837, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/4862d286b3f24812aa0c4fbc490a57e3, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5aa6cb8c4a4f48c0aa530466fea198dc, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9c97b82cb9734c909665d116147325b6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/eecd9766a802452e9d78751101bfec09, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/56b6e7d59d9d4f09a2c724bb7fa35c24, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/72a850dfeff440fb879b523727eeffa9, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/244ffe8d45e3487a8adf55c7f26ff093, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9badc4af24cf4fd886f1f7a71f5412ea, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/37e141db34134eb69d0aaac115388799, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/a8fa208494bf4b638f26787e8c4f2fc1, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/af6178a4ffe74dec8791794cd441ccd6, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3a527a664518437a9346d0bc6fb7c8de, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/e6c41d0ff9cb44c6a9501771fc212402, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0d53ced2cbe54e138eae68c41467296f] to archive 2023-04-24 09:01:17,041 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-24 09:01:17,042 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9ee13b1bd9e54876a36bd777132ac465 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9ee13b1bd9e54876a36bd777132ac465 2023-04-24 09:01:17,043 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0f6015dd8d3441ca881c36fdfbc4fb0f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0f6015dd8d3441ca881c36fdfbc4fb0f 2023-04-24 09:01:17,044 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/7535703bd9c14189be2b0fa353a4c1ad to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/7535703bd9c14189be2b0fa353a4c1ad 2023-04-24 09:01:17,045 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/04de6d7ee32247ffa1fdb48b61fffd4a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/04de6d7ee32247ffa1fdb48b61fffd4a 2023-04-24 09:01:17,046 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5f0b42148f064de49876dafee8697a62 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5f0b42148f064de49876dafee8697a62 2023-04-24 09:01:17,047 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/62be3ce3a7674888830178f5db28b47c to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/62be3ce3a7674888830178f5db28b47c 2023-04-24 09:01:17,048 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/2df0aea543394b70af88c5d1847da837 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/2df0aea543394b70af88c5d1847da837 2023-04-24 09:01:17,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/4862d286b3f24812aa0c4fbc490a57e3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/4862d286b3f24812aa0c4fbc490a57e3 2023-04-24 09:01:17,049 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5aa6cb8c4a4f48c0aa530466fea198dc to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/5aa6cb8c4a4f48c0aa530466fea198dc 2023-04-24 09:01:17,050 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9c97b82cb9734c909665d116147325b6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9c97b82cb9734c909665d116147325b6 2023-04-24 09:01:17,051 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/eecd9766a802452e9d78751101bfec09 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/eecd9766a802452e9d78751101bfec09 2023-04-24 09:01:17,052 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/56b6e7d59d9d4f09a2c724bb7fa35c24 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/56b6e7d59d9d4f09a2c724bb7fa35c24 2023-04-24 09:01:17,053 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/72a850dfeff440fb879b523727eeffa9 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/72a850dfeff440fb879b523727eeffa9 2023-04-24 09:01:17,053 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/244ffe8d45e3487a8adf55c7f26ff093 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/244ffe8d45e3487a8adf55c7f26ff093 2023-04-24 09:01:17,054 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9badc4af24cf4fd886f1f7a71f5412ea to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/9badc4af24cf4fd886f1f7a71f5412ea 2023-04-24 09:01:17,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/37e141db34134eb69d0aaac115388799 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/37e141db34134eb69d0aaac115388799 2023-04-24 09:01:17,055 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/a8fa208494bf4b638f26787e8c4f2fc1 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/a8fa208494bf4b638f26787e8c4f2fc1 2023-04-24 09:01:17,056 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/af6178a4ffe74dec8791794cd441ccd6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/af6178a4ffe74dec8791794cd441ccd6 2023-04-24 09:01:17,056 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3a527a664518437a9346d0bc6fb7c8de to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3a527a664518437a9346d0bc6fb7c8de 2023-04-24 09:01:17,057 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/e6c41d0ff9cb44c6a9501771fc212402 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/e6c41d0ff9cb44c6a9501771fc212402 2023-04-24 09:01:17,058 DEBUG [StoreCloser-TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0d53ced2cbe54e138eae68c41467296f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/0d53ced2cbe54e138eae68c41467296f 2023-04-24 09:01:17,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/recovered.edits/357.seqid, newMaxSeqId=357, maxSeqId=4 2023-04-24 09:01:17,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da. 2023-04-24 09:01:17,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for d6e3ad0fadc5b98d862d4692cd3586da: 2023-04-24 09:01:17,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(147): Closed d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:17,063 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=d6e3ad0fadc5b98d862d4692cd3586da, regionState=CLOSED 2023-04-24 09:01:17,063 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682326877063"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682326877063"}]},"ts":"1682326877063"} 2023-04-24 09:01:17,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-24 09:01:17,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure d6e3ad0fadc5b98d862d4692cd3586da, server=jenkins-hbase16.apache.org,42837,1682326693550 in 1.9130 sec 2023-04-24 09:01:17,066 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-24 09:01:17,066 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d6e3ad0fadc5b98d862d4692cd3586da, UNASSIGN in 1.9180 sec 2023-04-24 09:01:17,066 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682326877066"}]},"ts":"1682326877066"} 2023-04-24 09:01:17,067 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-24 09:01:17,096 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-24 09:01:17,099 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9730 sec 2023-04-24 09:01:17,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-24 09:01:17,239 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-24 09:01:17,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete TestAcidGuarantees 2023-04-24 09:01:17,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:01:17,245 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:01:17,246 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:01:17,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-24 09:01:17,254 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:17,259 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C, FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/recovered.edits] 2023-04-24 09:01:17,264 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7a3a58b6492a416080e5021dda85b5bd to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/7a3a58b6492a416080e5021dda85b5bd 2023-04-24 09:01:17,266 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ec3b0cae57da4816ae42a80bc05bb44a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/A/ec3b0cae57da4816ae42a80bc05bb44a 2023-04-24 09:01:17,270 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/39ba5f704e4d4caab596e02363374750 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/39ba5f704e4d4caab596e02363374750 2023-04-24 09:01:17,272 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/730c26ee91a445338c466f2f95c6745b to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/B/730c26ee91a445338c466f2f95c6745b 2023-04-24 09:01:17,275 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3606fdc7bf3f466f8a0e009dfacd8475 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/3606fdc7bf3f466f8a0e009dfacd8475 2023-04-24 09:01:17,277 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/f29e04b16c9547cb8c5b34f7d05a84a5 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/C/f29e04b16c9547cb8c5b34f7d05a84a5 2023-04-24 09:01:17,281 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/recovered.edits/357.seqid to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da/recovered.edits/357.seqid 2023-04-24 09:01:17,282 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/.tmp/data/default/TestAcidGuarantees/d6e3ad0fadc5b98d862d4692cd3586da 2023-04-24 09:01:17,282 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-24 09:01:17,283 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 09:01:17,284 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-24 09:01:17,287 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240bfc169decc74e4183356fe68da2b0b0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304240bfc169decc74e4183356fe68da2b0b0 2023-04-24 09:01:17,289 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042421063f95286d4b21a76bb5a2107c13b6 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042421063f95286d4b21a76bb5a2107c13b6 2023-04-24 09:01:17,291 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042444ddf223875349f0a8f318a7218a7618 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042444ddf223875349f0a8f318a7218a7618 2023-04-24 09:01:17,292 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042445318fe8b0e24d04b6c8f1d2d9a1ec59 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042445318fe8b0e24d04b6c8f1d2d9a1ec59 2023-04-24 09:01:17,292 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042448746393b7c84362bfa96b69e32d6f65 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042448746393b7c84362bfa96b69e32d6f65 2023-04-24 09:01:17,293 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245c46d3ee93fa4d08950a17691b395c1a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304245c46d3ee93fa4d08950a17691b395c1a 2023-04-24 09:01:17,294 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042467701ed80d1b4fc981e8f8aea4ffa89a to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042467701ed80d1b4fc981e8f8aea4ffa89a 2023-04-24 09:01:17,294 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042492d50e51c64941209b3267049b938b5e to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042492d50e51c64941209b3267049b938b5e 2023-04-24 09:01:17,295 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bac07b638ffe4cbf97ab2524d570379f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424bac07b638ffe4cbf97ab2524d570379f 2023-04-24 09:01:17,296 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c9459e1066104c388ed74e514cc0cdd4 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424c9459e1066104c388ed74e514cc0cdd4 2023-04-24 09:01:17,296 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d187929e036941e387e1bb8ae7917cf3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424d187929e036941e387e1bb8ae7917cf3 2023-04-24 09:01:17,297 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424dc363da8b09f4dfd80fb0dd70598db53 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424dc363da8b09f4dfd80fb0dd70598db53 2023-04-24 09:01:17,298 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e58aa4b1649441fdb3da68e279cd39ef to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e58aa4b1649441fdb3da68e279cd39ef 2023-04-24 09:01:17,298 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e7db7e776a974b1582d7ea4dba67ea9f to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424e7db7e776a974b1582d7ea4dba67ea9f 2023-04-24 09:01:17,299 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ec4ec137765b4c0f92080e0250706fc0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ec4ec137765b4c0f92080e0250706fc0 2023-04-24 09:01:17,300 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f9acca6720fe456d993fc12fe9e7c5c3 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424f9acca6720fe456d993fc12fe9e7c5c3 2023-04-24 09:01:17,300 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fc6ea0f54ad149a89d4ace5451f8ccec to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424fc6ea0f54ad149a89d4ace5451f8ccec 2023-04-24 09:01:17,301 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ff7f82bdddd4416bab4fff10ac004fa0 to hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230424ff7f82bdddd4416bab4fff10ac004fa0 2023-04-24 09:01:17,302 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-24 09:01:17,303 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:01:17,304 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-24 09:01:17,306 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-24 09:01:17,306 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:01:17,307 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-24 09:01:17,307 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682326877307"}]},"ts":"9223372036854775807"} 2023-04-24 09:01:17,308 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-24 09:01:17,308 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d6e3ad0fadc5b98d862d4692cd3586da, NAME => 'TestAcidGuarantees,,1682326844478.d6e3ad0fadc5b98d862d4692cd3586da.', STARTKEY => '', ENDKEY => ''}] 2023-04-24 09:01:17,308 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-24 09:01:17,308 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682326877308"}]},"ts":"9223372036854775807"} 2023-04-24 09:01:17,309 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-24 09:01:17,337 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-24 09:01:17,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 97 msec 2023-04-24 09:01:17,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40429] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-24 09:01:17,349 INFO [Listener at localhost.localdomain/44291] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-24 09:01:17,364 INFO [Listener at localhost.localdomain/44291] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 230), OpenFileDescriptor=520 (was 547), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=301 (was 339), ProcessCount=165 (was 165), AvailableMemoryMB=8486 (was 8388) - AvailableMemoryMB LEAK? - 2023-04-24 09:01:17,365 INFO [Listener at localhost.localdomain/44291] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-24 09:01:17,365 INFO [Listener at localhost.localdomain/44291] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-24 09:01:17,365 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7256a797 to 127.0.0.1:63978 2023-04-24 09:01:17,365 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:17,366 DEBUG [Listener at localhost.localdomain/44291] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-24 09:01:17,366 DEBUG [Listener at localhost.localdomain/44291] util.JVMClusterUtil(257): Found active master hash=244386752, stopped=false 2023-04-24 09:01:17,366 INFO [Listener at localhost.localdomain/44291] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 09:01:17,379 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-24 09:01:17,379 INFO [Listener at localhost.localdomain/44291] procedure2.ProcedureExecutor(629): Stopping 2023-04-24 09:01:17,379 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-24 09:01:17,379 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 09:01:17,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-24 09:01:17,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-24 09:01:17,380 DEBUG [Listener at localhost.localdomain/44291] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74692548 to 127.0.0.1:63978 2023-04-24 09:01:17,380 DEBUG [Listener at localhost.localdomain/44291] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:17,380 INFO [Listener at localhost.localdomain/44291] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,42837,1682326693550' ***** 2023-04-24 09:01:17,380 INFO [Listener at localhost.localdomain/44291] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-24 09:01:17,381 INFO [RS:0;jenkins-hbase16:42837] regionserver.HeapMemoryManager(220): Stopping 2023-04-24 09:01:17,381 INFO [RS:0;jenkins-hbase16:42837] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-24 09:01:17,381 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-24 09:01:17,381 INFO [RS:0;jenkins-hbase16:42837] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-24 09:01:17,381 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(3304): Received CLOSE for 81147daa64e17749d4bc895e5511dda1 2023-04-24 09:01:17,382 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:17,382 DEBUG [RS:0;jenkins-hbase16:42837] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c53079d to 127.0.0.1:63978 2023-04-24 09:01:17,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 81147daa64e17749d4bc895e5511dda1, disabling compactions & flushes 2023-04-24 09:01:17,382 DEBUG [RS:0;jenkins-hbase16:42837] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:17,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 09:01:17,383 INFO [RS:0;jenkins-hbase16:42837] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-24 09:01:17,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 09:01:17,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. after waiting 0 ms 2023-04-24 09:01:17,383 INFO [RS:0;jenkins-hbase16:42837] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-24 09:01:17,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 09:01:17,383 INFO [RS:0;jenkins-hbase16:42837] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-24 09:01:17,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 81147daa64e17749d4bc895e5511dda1 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-24 09:01:17,383 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-24 09:01:17,384 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-24 09:01:17,384 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 81147daa64e17749d4bc895e5511dda1=hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1.} 2023-04-24 09:01:17,384 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-24 09:01:17,385 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-24 09:01:17,385 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-24 09:01:17,385 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-24 09:01:17,385 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-24 09:01:17,385 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-24 09:01:17,387 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1505): Waiting on 1588230740, 81147daa64e17749d4bc895e5511dda1 2023-04-24 09:01:17,396 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/.tmp/info/0d694c2f5d4548849de3d004d2c57a05 2023-04-24 09:01:17,401 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0d694c2f5d4548849de3d004d2c57a05 2023-04-24 09:01:17,415 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-24 09:01:17,588 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1505): Waiting on 1588230740, 81147daa64e17749d4bc895e5511dda1 2023-04-24 09:01:17,788 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1505): Waiting on 1588230740, 81147daa64e17749d4bc895e5511dda1 2023-04-24 09:01:17,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1/.tmp/info/c49f2100849342719f2b414a7163d595 2023-04-24 09:01:17,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1/.tmp/info/c49f2100849342719f2b414a7163d595 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1/info/c49f2100849342719f2b414a7163d595 2023-04-24 09:01:17,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1/info/c49f2100849342719f2b414a7163d595, entries=2, sequenceid=6, filesize=4.8 K 2023-04-24 09:01:17,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 81147daa64e17749d4bc895e5511dda1 in 420ms, sequenceid=6, compaction requested=false 2023-04-24 09:01:17,806 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/.tmp/rep_barrier/c6201b62de704e16ad7e6387a5b24daf 2023-04-24 09:01:17,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/namespace/81147daa64e17749d4bc895e5511dda1/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-24 09:01:17,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 09:01:17,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 81147daa64e17749d4bc895e5511dda1: 2023-04-24 09:01:17,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682326696232.81147daa64e17749d4bc895e5511dda1. 2023-04-24 09:01:17,810 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c6201b62de704e16ad7e6387a5b24daf 2023-04-24 09:01:17,988 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-24 09:01:18,189 DEBUG [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-24 09:01:18,198 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-24 09:01:18,198 INFO [regionserver/jenkins-hbase16:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-24 09:01:18,216 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/.tmp/table/a859b2ca9fb048a2b7861615d75321f0 2023-04-24 09:01:18,225 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a859b2ca9fb048a2b7861615d75321f0 2023-04-24 09:01:18,226 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/.tmp/info/0d694c2f5d4548849de3d004d2c57a05 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/info/0d694c2f5d4548849de3d004d2c57a05 2023-04-24 09:01:18,231 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0d694c2f5d4548849de3d004d2c57a05 2023-04-24 09:01:18,231 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/info/0d694c2f5d4548849de3d004d2c57a05, entries=22, sequenceid=93, filesize=7.1 K 2023-04-24 09:01:18,234 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/.tmp/rep_barrier/c6201b62de704e16ad7e6387a5b24daf as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/rep_barrier/c6201b62de704e16ad7e6387a5b24daf 2023-04-24 09:01:18,238 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c6201b62de704e16ad7e6387a5b24daf 2023-04-24 09:01:18,239 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/rep_barrier/c6201b62de704e16ad7e6387a5b24daf, entries=6, sequenceid=93, filesize=5.5 K 2023-04-24 09:01:18,239 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/.tmp/table/a859b2ca9fb048a2b7861615d75321f0 as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/table/a859b2ca9fb048a2b7861615d75321f0 2023-04-24 09:01:18,244 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a859b2ca9fb048a2b7861615d75321f0 2023-04-24 09:01:18,244 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/table/a859b2ca9fb048a2b7861615d75321f0, entries=9, sequenceid=93, filesize=5.5 K 2023-04-24 09:01:18,245 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 860ms, sequenceid=93, compaction requested=false 2023-04-24 09:01:18,256 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-24 09:01:18,256 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-24 09:01:18,258 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-24 09:01:18,258 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-24 09:01:18,258 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-24 09:01:18,389 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,42837,1682326693550; all regions closed. 2023-04-24 09:01:18,396 DEBUG [RS:0;jenkins-hbase16:42837] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/oldWALs 2023-04-24 09:01:18,396 INFO [RS:0;jenkins-hbase16:42837] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C42837%2C1682326693550.meta:.meta(num 1682326695966) 2023-04-24 09:01:18,400 DEBUG [RS:0;jenkins-hbase16:42837] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/oldWALs 2023-04-24 09:01:18,400 INFO [RS:0;jenkins-hbase16:42837] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C42837%2C1682326693550:(num 1682326695352) 2023-04-24 09:01:18,400 DEBUG [RS:0;jenkins-hbase16:42837] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:18,400 INFO [RS:0;jenkins-hbase16:42837] regionserver.LeaseManager(133): Closed leases 2023-04-24 09:01:18,401 INFO [RS:0;jenkins-hbase16:42837] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-24 09:01:18,401 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-24 09:01:18,402 INFO [RS:0;jenkins-hbase16:42837] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:42837 2023-04-24 09:01:18,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-24 09:01:18,422 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,42837,1682326693550 2023-04-24 09:01:18,422 ERROR [Listener at localhost.localdomain/44291-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5cbd9de8 rejected from java.util.concurrent.ThreadPoolExecutor@588212a7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1898] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 09:01:18,423 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-24 09:01:18,423 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,42837,1682326693550] 2023-04-24 09:01:18,423 ERROR [Listener at localhost.localdomain/44291-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c35e175 rejected from java.util.concurrent.ThreadPoolExecutor@588212a7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1898] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 09:01:18,424 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,42837,1682326693550; numProcessing=1 2023-04-24 09:01:18,437 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,42837,1682326693550 already deleted, retry=false 2023-04-24 09:01:18,438 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,42837,1682326693550 expired; onlineServers=0 2023-04-24 09:01:18,438 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase16.apache.org,40429,1682326692433' ***** 2023-04-24 09:01:18,438 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-24 09:01:18,438 DEBUG [M:0;jenkins-hbase16:40429] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fac01fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-04-24 09:01:18,438 INFO [M:0;jenkins-hbase16:40429] regionserver.HRegionServer(1145): stopping server jenkins-hbase16.apache.org,40429,1682326692433 2023-04-24 09:01:18,438 INFO [M:0;jenkins-hbase16:40429] regionserver.HRegionServer(1171): stopping server jenkins-hbase16.apache.org,40429,1682326692433; all regions closed. 2023-04-24 09:01:18,438 DEBUG [M:0;jenkins-hbase16:40429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-24 09:01:18,438 DEBUG [M:0;jenkins-hbase16:40429] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-24 09:01:18,438 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-24 09:01:18,438 DEBUG [M:0;jenkins-hbase16:40429] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-24 09:01:18,438 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1682326695051] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1682326695051,5,FailOnTimeoutGroup] 2023-04-24 09:01:18,438 INFO [M:0;jenkins-hbase16:40429] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-24 09:01:18,438 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1682326695051] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1682326695051,5,FailOnTimeoutGroup] 2023-04-24 09:01:18,438 INFO [M:0;jenkins-hbase16:40429] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-24 09:01:18,438 INFO [M:0;jenkins-hbase16:40429] hbase.ChoreService(369): Chore service for: master/jenkins-hbase16:0 had [] on shutdown 2023-04-24 09:01:18,439 DEBUG [M:0;jenkins-hbase16:40429] master.HMaster(1512): Stopping service threads 2023-04-24 09:01:18,439 INFO [M:0;jenkins-hbase16:40429] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-24 09:01:18,439 ERROR [M:0;jenkins-hbase16:40429] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-24 09:01:18,440 INFO [M:0;jenkins-hbase16:40429] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-24 09:01:18,440 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-24 09:01:18,446 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-24 09:01:18,446 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-24 09:01:18,446 DEBUG [M:0;jenkins-hbase16:40429] zookeeper.ZKUtil(398): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-24 09:01:18,446 WARN [M:0;jenkins-hbase16:40429] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-24 09:01:18,446 INFO [M:0;jenkins-hbase16:40429] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-24 09:01:18,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-24 09:01:18,446 INFO [M:0;jenkins-hbase16:40429] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-24 09:01:18,447 DEBUG [M:0;jenkins-hbase16:40429] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-24 09:01:18,447 INFO [M:0;jenkins-hbase16:40429] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 09:01:18,447 DEBUG [M:0;jenkins-hbase16:40429] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 09:01:18,447 DEBUG [M:0;jenkins-hbase16:40429] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-24 09:01:18,447 DEBUG [M:0;jenkins-hbase16:40429] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 09:01:18,447 INFO [M:0;jenkins-hbase16:40429] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.31 KB heapSize=393.95 KB 2023-04-24 09:01:18,472 INFO [M:0;jenkins-hbase16:40429] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.31 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/591fd225d075448eaa277789614d3cda 2023-04-24 09:01:18,475 INFO [M:0;jenkins-hbase16:40429] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 591fd225d075448eaa277789614d3cda 2023-04-24 09:01:18,476 DEBUG [M:0;jenkins-hbase16:40429] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/591fd225d075448eaa277789614d3cda as hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/591fd225d075448eaa277789614d3cda 2023-04-24 09:01:18,479 INFO [M:0;jenkins-hbase16:40429] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 591fd225d075448eaa277789614d3cda 2023-04-24 09:01:18,479 INFO [M:0;jenkins-hbase16:40429] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46389/user/jenkins/test-data/7b03821a-529e-6165-7266-7bdeec7de42f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/591fd225d075448eaa277789614d3cda, entries=65, sequenceid=578, filesize=24.7 K 2023-04-24 09:01:18,480 INFO [M:0;jenkins-hbase16:40429] regionserver.HRegion(2948): Finished flush of dataSize ~345.31 KB/353595, heapSize ~393.94 KB/403392, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=578, compaction requested=false 2023-04-24 09:01:18,484 INFO [M:0;jenkins-hbase16:40429] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-24 09:01:18,484 DEBUG [M:0;jenkins-hbase16:40429] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-24 09:01:18,487 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-24 09:01:18,487 INFO [M:0;jenkins-hbase16:40429] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-24 09:01:18,488 INFO [M:0;jenkins-hbase16:40429] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:40429 2023-04-24 09:01:18,496 DEBUG [M:0;jenkins-hbase16:40429] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase16.apache.org,40429,1682326692433 already deleted, retry=false 2023-04-24 09:01:18,529 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-24 09:01:18,529 INFO [RS:0;jenkins-hbase16:42837] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,42837,1682326693550; zookeeper connection closed. 2023-04-24 09:01:18,530 ERROR [Listener at localhost.localdomain/44291-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@648845d1 rejected from java.util.concurrent.ThreadPoolExecutor@588212a7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1898] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 09:01:18,530 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): regionserver:42837-0x101ba998db50001, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-24 09:01:18,530 ERROR [Listener at localhost.localdomain/44291-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4e220015 rejected from java.util.concurrent.ThreadPoolExecutor@588212a7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1898] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 09:01:18,530 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@60d4d7c2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@60d4d7c2 2023-04-24 09:01:18,530 INFO [Listener at localhost.localdomain/44291] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-24 09:01:18,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-24 09:01:18,630 INFO [M:0;jenkins-hbase16:40429] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase16.apache.org,40429,1682326692433; zookeeper connection closed. 2023-04-24 09:01:18,630 ERROR [Listener at localhost.localdomain/44291-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1be7f066 rejected from java.util.concurrent.ThreadPoolExecutor@59ea4db0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3184] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 09:01:18,630 DEBUG [Listener at localhost.localdomain/44291-EventThread] zookeeper.ZKWatcher(600): master:40429-0x101ba998db50000, quorum=127.0.0.1:63978, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-24 09:01:18,630 ERROR [Listener at localhost.localdomain/44291-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6d225852 rejected from java.util.concurrent.ThreadPoolExecutor@59ea4db0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3184] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-24 09:01:18,631 WARN [Listener at localhost.localdomain/44291] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-24 09:01:18,635 INFO [Listener at localhost.localdomain/44291] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-24 09:01:18,744 WARN [BP-518531597-188.40.62.62-1682326689016 heartbeating to localhost.localdomain/127.0.0.1:46389] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-24 09:01:18,744 WARN [BP-518531597-188.40.62.62-1682326689016 heartbeating to localhost.localdomain/127.0.0.1:46389] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-518531597-188.40.62.62-1682326689016 (Datanode Uuid 23f6c51a-5397-46cb-8527-136c6ed29987) service to localhost.localdomain/127.0.0.1:46389 2023-04-24 09:01:18,748 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/cluster_34f00b0f-ffba-bb69-e77b-b55f002d7f8b/dfs/data/data1/current/BP-518531597-188.40.62.62-1682326689016] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-24 09:01:18,749 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ecbf45d3-7681-a17c-b495-28c139692fb5/cluster_34f00b0f-ffba-bb69-e77b-b55f002d7f8b/dfs/data/data2/current/BP-518531597-188.40.62.62-1682326689016] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-24 09:01:18,782 INFO [Listener at localhost.localdomain/44291] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-24 09:01:18,937 INFO [Listener at localhost.localdomain/44291] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-24 09:01:19,096 INFO [Listener at localhost.localdomain/44291] hbase.HBaseTestingUtility(1293): Minicluster is down